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Abstract 


The  Standard  CMMI  Appraisal  Method  for  Process  Improvement  (SCAMPI)  is  designed  to 
provide  benchmark  quality  ratings  relative  to  Capability  Maturity  Model  Integration  (CMMI) 
models  and  the  People  CMM.  The  SCAMPI  Method  Definition  Document  (MDD)  describes  the 
requirements,  activities,  and  practices  associated  with  the  processes  that  compose  the  SCAMPI 
method.  The  MDD  also  contains  precise  descriptions  of  the  method’s  context,  concepts,  and 
architecture. 
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About  this  Document 


This  document,  also  called  the  Method  Definition  Document  (MDD)  describes  the  Class  A 
Standard  CMMI  Appraisal  Method  for  Process  Improvement  (SCAMPI).  The  MDD  is  divided 
into  three  major  parts,  each  with  a  different  level  of  detail,  intended  usage,  and  primary  audience. 
The  structure,  audiences,  and  suggested  use  of  each  part  of  the  document  are  described  below. 

Revision  History 

April  12,  201 1  SCAMPI  Method  Definition  Document  published  to  the  SEI  website  at  the 
following  URL:  <http://www.sei.cmu.edu/library/abstracts/reports/llhb001.cfm> 

April  25,  201 1  -  SCAMPI  Method  Definition  Document  republished  to  the  SEI  website  with 
corrections. 

June  1,  201 1-  SCAMPI  Method  Definition  Document  republished  to  the  SEI  website  with 
corrections. 

For  details  about  revisions  and  editorial  corrections,  please  reference  the  Revision  sheet  at  the 
following  URL:  http://www.sei.cmu.edu/cmmi/tools/upload/llhb001-revisions.cfm 

Part  I:  Overview 

Part  I  of  the  MDD  provides  an  overview  of  the  method’s  context,  concepts,  and  architecture.  Part 
I  gives  a  big  picture  of  the  method,  rather  than  details  about  how  to  enact  it.  Table  1  shows  the 
contents  of  Part  I  of  the  MDD. 

Table  1:  Part  I  Contents 


Section 

Page  Number 

About  This  Document 

3 

Executive  Summary 

9 

SCAMPI  A  Method  Overview 

13 

Part  II:  Process  Definitions 

Part  II  of  the  MDD  describes  the  method  requirements  and  the  detailed  activities  and  practices 
associated  with  each  of  the  processes  that  compose  the  SCAMPI  A  method.  Part  II  lists  required 
practices,  parameters,  and  the  limits  of  allowable  variation,  and  gives  guidance  for  enacting  the 
method.  Table  2  shows  the  contents  of  Part  II. 
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Table  2:  Part  II  Contents 


Phase 

Process 

Page  Number 

1 :  Plan  and  Prepare  for 
Appraisal 

1.1  Analyze  Requirements 

37 

1 .2  Develop  Appraisal  Plan 

59 

1.3  Select  and  Prepare  Team 

76 

1.4  Obtain  and  Inventory  Initial  Objective  Evidence 

96 

1.5  Prepare  for  Appraisal  Conduct 

104 

2:  Conduct  Appraisal 

2.1  Prepare  Participants 

113 

2.2  Examine  Objective  Evidence 

116 

2.3  Document  Objective  Evidence 

128 

2.4  Verify  Objective  Evidence 

139 

2.5  Validate  Preliminary  Findings 

149 

2.6  Generate  Appraisal  Results 

155 

3:  Report  Results 

3.1  Deliver  Appraisal  Results 

165 

3.2  Package  and  Archive  Appraisal  Assets 

173 

Part  ill:  Appendices,  References,  and  Glossary 

Part  III  of  the  MDD  includes  appendices  that  elaborate  selected  topics  and  supplement  the  first 
two  parts  of  this  document.  Read  the  first  two  parts  of  the  MDD  prior  to  reading  Part  III.  The 
topical  elaboration  and  reference  material  available  in  the  appendices  provides  deeper  insights  to 
readers  already  knowledgeable  about  the  material.  Table  3  shows  the  contents  of  Part  III. 

Table  3:  Part  III  Contents 


Section 

Page  Number 

Appendix  A 

The  Role  of  Objective  Evidence  in  Verifying  Practice  Implementation 

185 

Appendix  B 

Alternative  Practice  Identification  and  Characterization  Guidance 

191 

Appendix  C 

Roles  and  Responsibilities 

195 

Appendix  D 

Reporting  Requirements  and  Options 

201 

Appendix  E 

Managed  Discovery 

203 

Appendix  F 

Scoping  and  Sampling  in  SCAMPI  A  Appraisals 

209 

Appendix  G 

SCAMPI  A  Appraisals  Including  Multiple  Models 

241 

Appendix  H 

SCAMPI  A  Tailoring  Checklist 

247 

References/Bibliography 

249 

Glossary 

251 
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Audiences  for  this  Document 

The  MDD  is  primarily  intended  for  SCAMPI  Lead  Appraisers  certified  by  the  Software 
Engineering  Institute  (SEI).  It  is  expected  that  these  professionals  have  the  prerequisite 
knowledge  and  skills  specified  by  the  SEI  Appraisal  program  (see  http://www.sei.cmu.edu/  for 
details),  and  that  they  use  the  MDD  as  a  key  part  of  their  knowledge  infrastructure.  SCAMPI 
Lead  Appraisers  are  the  primary  audience  for  Part  II.  The  MDD  is  also  used  as  a  training  aid  in 
SCAMPI  Lead  Appraiser  training. 

Appraisal  team  members  are  expected  to  refer  to  this  document  as  a  training  aid.  Portions  of  the 
MDD  may  also  be  used  as  work  aids  during  the  conduct  of  an  appraisal.  Potential  appraisal  team 
members  can  use  the  MDD  to  build  their  knowledge  base  so  they  can  participate  in  a  future 
appraisal. 

Appraisal  stakeholders  are  also  part  of  the  targeted  audience  for  the  MDD,  particularly  for  Part  I. 
These  stakeholders  include  the  following: 

•  appraisal  sponsors  leaders  who  sponsor  appraisals  to  meet  business  objectives 

•  process  group  members-process  improvement  specialists  who  need  to  understand  the  method, 
and  sometimes  to  also  help  others  gain  familiarity  with  the  method 

•  other  interested  people-those  who  want  deeper  insight  into  the  methodology  for  purposes 
such  as  ensuring  that  they  have  an  informed  basis  for  interpreting  SCAMPI  A  outputs  or 
making  comparisons  among  similar  methodologies 

How  to  Use  this  Document 

Part  I 

It  is  expected  that  every  member  of  the  audience  for  this  document  will  find  value  in  Part  I.  The 
two  primary  sections  in  this  part  are  the  Executive  Summary  and  the  Method  Overview. 

The  Executive  Summary  is  intended  to  provide  high-level  information  describing  SCAMPI  A, 
and  does  not  require  extensive  knowledge  of  appraisals.  This  portion  of  the  document  may  be 
excerpted  and  provided  to  a  more  casual  reader  or  a  stakeholder  in  need  of  general  information  to 
support  their  decision  to  conduct  an  appraisal. 

The  Method  Overview  section  provides  comprehensive  coverage  of  SCAMPI  A,  and  can  be  used 
to  begin  building  a  base  of  knowledge  for  readers  who  need  more  detailed  infonnation.  Appraisal 
sponsors  wanting  more  than  a  summary  view  should  read  this  section.  Every  prospective 
SCAMPI  A  appraisal  team  leader  and  team  member  is  expected  to  read  this  section  of  the 
document  to  ensure  that  they  have  the  “big  picture”  before  they  study  the  detailed  methodology. 
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Part  II 


People  who  will  enact  an  appraisal  are  expected  to  read  the  second  part  of  the  document. 

Members  of  this  audience  need  to  know  how  to  enact  the  method,  not  just  what  the  method  is. 

Part  II  is  divided  into  Process  Definitions,  which  are  in  turn  divided  into  Activities.  Each  Activity 
delineates  Required  Practices,  Parameters  and  Limits,  and  Implementation  Guidance. 

There  are  several  processes  contained  in  SCAMPI  A.  The  processes  support  a  variety  of 
orderings  and  enactments  to  facilitate  a  variety  of  usage  modes  for  SCAMPI  A.  The  temporal 
flow,  as  well  as  the  flow  of  inputs  and  outputs  among  the  processes,  is  described  in  the  Method 
Overview  section.  The  Process  Definitions  are  not  intended  to  provide  a  start-to-fmish  view  of 
SCAMPI  A.  Instead,  these  sections  provide  detailed  definitions  of  processes  and  activities  that  are 
implemented  according  to  the  appraisal  plan  created  by  the  appraisal  team  leader. 

Each  of  the  Process  Definitions  begins  with  an  overview  of  the  process.  Every  process  is  defined 
by  information  contained  in  the  elements  shown  in  Table  4. 


Table  4:  Process  Definition  Elements 


Element 

Description 

Purpose 

A  brief  summary  of  what  is  accomplished  by  enacting  the  process 

Entry  Criteria 

Conditions  that  must  be  met  before  enacting  the  process 

Inputs 

Artifacts  or  information  needed  to  enact  the  process 

Activities 

The  set  of  actions  which,  in  combination,  make  up  the  process 

Outputs 

Artifacts  and  assets  that  result  from  enacting  the  process 

Outcome 

Any  change  in  important  conditions  or  artifacts  that  results  from 
enacting  the  process 

Exit  Criteria 

Conditions  to  be  met  before  the  process  can  be  considered  complete 

Key  Points 

A  summary  of  the  most  notable  events  associated  with  the  process 

Tools  and  Techniques 

Work  aids  commonly  used  in  enacting  the  process 

Metrics 

Useful  measures  that  support  the  process  enactment,  or  future 
enactments 

Verification  and 

Validation 

Techniques  to  verify  and/or  validate  the  enactment  of  the  process 

Records 

Information  to  be  retained  for  future  use 

Interfaces  with  Other  Processes 

A  discussion  of  how  the  process  interacts  with  other  processes  in  the 
method 

Summary  of  Activities 

A  narrative  summary  of  the  set  of  activities 
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Following  the  introductory  material,  each  Activity  that  is  a  part  of  the  Process  Definition  is 
briefly  summarized  to  orient  the  reader  to  the  scope  of  the  activity.  Each  Activity  includes  the 
elements  shown  in  Table  5. 


Table  5:  Activity  Elements 


Element 

Description 

Activity  Description 

A  brief  overview  of  what  is  accomplished  by  enacting  the  activity 

Required  Practices 

A  listing  of  practices  that  must  be  implemented  to  consider  the  enactment 
a  valid  SCAMPI  A 

Parameters  and  Limits 

Acceptable  limits  for  things  that  are  allowed  to  vary,  and  acceptable  limits 
for  things  under  the  discretion  of  the  appraisal  team  leader 

Implementation  Guidance 

A  narrative  description  of  advice  or  things  to  consider  in  performing  the 
activity 

Part  II  provides  complete,  unambiguous  descriptions  of  the  method  processes  and  activities.  In 
combination  with  the  training  materials  and  work  aids  that  comprise  the  SEI’s  appraisal  program, 
this  information  provides  a  firm  basis  for  standardization  of  the  practice  of  Process  Appraisals. 


Part  III 

The  appendices  of  the  document  provide  detailed  coverage  of  special  topics  as  well  as  references 
and  a  glossary.  Readers  knowledgeable  about  SCAMPI  A  are  expected  to  read  these  sections  to 
gain  further  understanding  of  best  practices  and  implementation  guidance  on  SCAMPI  A  concepts 
that  may  span  multiple  appraisal  activities. 

Feedback  Information 

We  are  very  interested  in  your  ideas  for  improving  this  document.  See  the  CMMI  web  site  for 
information  on  how  to  provide  feedback  by  using  a  Change  Request  fonn: 
http://www.sei.cmu.edu/cmmi/tools/cr/ 

If  you  have  questions,  please  send  an  email  to  cmmi-comments@sei.cmu.edu. 
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Executive  Summary 


What  Is  SCAMPI  A? 

The  Standard  CMMI  Appraisal  Method  for  Process  Improvement  (SCAMPI)  is  designed  to 
provide  benchmark-quality  ratings  relative  to  Capability  Maturity  Model  Integration  (CMMI) 
models.  SCAMPI  A  satisfies  all  of  the  Appraisal  Requirements  for  CMMI  (ARC)  requirements 
for  a  Class  A  appraisal  method.  Although  designed  for  conducting  appraisals  against  CMMI- 
based  reference  models,  the  SCAMPI  A  method  can  also  be  applied  for  conducting  appraisals 
against  the  People  CMM  and  other  reference  models. 

SCAMPI  A  enables  a  sponsor  to 

•  gain  insight  into  an  organization’s  capability  by  identifying  the  strengths  and  weaknesses  of 
its  current  processes  relative  to  appraisal  reference  model(s) 

•  prioritize  improvement  plans 

•  focus  on  improvements  (correct  weaknesses  that  generate  risks)  that  are  most  beneficial  to  the 
organization  given  its  current  level  of  organizational  maturity  or  process  capabilities 

•  derive  capability  level  ratings  as  well  as  a  maturity  level  rating 

•  identify  risks  relative  to  capability/maturity  determinations 

As  a  Class  A  appraisal  method,  SCAMPI  A  is  an  appropriate  tool  for  benchmarking.  Sponsors 
who  want  to  compare  an  organization’s  process  improvement  achievements  with  other 
organizations  in  the  industry  may  have  a  maturity  level  determined  as  part  of  the  appraisal 
process. 

Decisions  made  on  the  basis  of  maturity  level  ratings  are  only  valid  if  the  ratings  are  based  on 
known  criteria.  Consequently,  contextual  information — organizational  unit,  organizational  scope, 
reference  model  scope,  appraisal  method  type,  the  identity  of  the  appraisal  team  leader  and  the 
team — are  items  for  which  criteria  and  guidance  are  provided  within  the  method  to  ensure  a 
consistent  interpretation  within  the  community.  Benchmarking  can  only  be  valid  when  there  is  a 
consistent  basis  for  establishing  the  benchmarks. 

The  SEI  maintains  industry  aggregates  for  appraisal  results.  These  data  are  reported  in  industry 
maturity  profiles  gathered  from  organizations  that  have  performed  appraisals  since  1987.  The 
profile  is  based  on  appraisal  data  provided  by  SEI-trained  professionals,  and  is  updated  twice 
annually. 

The  SEI  supports  the  SCAMPI  A  method  and  operates  a  certification  program  for  SCAMPI  Lead 
Appraisers.  Additional  details  can  be  found  on  the  SEI  web  site  at  http://www.sei.cmu.edu. 


CMU/SEI-2011-HB-001  |  9 


Core  Concepts  and  Approach 

SCAMPI  A,  as  a  benchmarking  appraisal  method,  relies  on  an  aggregation  of  information  that  is 
collected  via  defined  types  of  objective  evidence.  The  objective  evidence  feeds  an  “information¬ 
processing  engine”  whose  parts  are  made  up  of  a  series  of  data  transformations.  The  appraisal 
team  observes,  hears,  and  reads  infonnation  that  is  transformed  into  notes,  and  then  into 
characterizations  of  practice  implementation  gaps  or  compliance,  and  then  into  preliminary 
findings.  These  findings  are  validated  by  the  organizational  unit  before  they  become  final 
findings.  The  critical  concept  is  that  these  transformations  are  applied  to  data  reflecting  the 
enacted  processes  in  the  organizational  unit  and  the  appraisal  reference  model,  and  this  collection 
of  data  forms  the  basis  for  ratings  and  other  appraisal  results. 

Planning  is  critical  to  the  execution  of  SCAMPI  A.  All  phase  and  process  activities  briefly 
discussed  below  derive  from  a  well-articulated  plan  developed  by  the  appraisal  team  leader  in 
concert  with  members  of  the  appraised  organization  and  the  appraisal  sponsor. 

SCAMPI  A  Methodology 

SCAMPI  A  consists  of  three  phases  and  several  essential  processes,  as  was  shown  in  Table  2. 
Each  phase  is  described  in  detail  below. 

Phase  1 :  Plan  and  Prepare  for  Appraisal 

Appraisal  planning  starts  with  understanding  the  sponsor’s  objectives,  requirements,  and 
constraints.  All  other  planning,  preparation,  execution,  and  reporting  of  results  proceed  from  this 
initial  activity.  Because  of  the  significant  investment  and  logistical  planning  involved, 
considerable  iteration  and  refinement  of  planning  activities  should  be  expected  in  phase  1.  With 
each  subsequent  phase,  the  amount  of  iteration  will  decrease  as  data  are  collected,  analyzed, 
refined,  and  translated  into  findings  of  significance  relative  to  the  model. 

A  team  of  experienced  and  trained  personnel  performs  a  SCAMPI  A  over  a  period  of  time 
negotiated  by  the  sponsor  and  the  appraisal  team  leader.  The  scope  of  the  organization  to  be 
appraised,  as  well  as  the  scope  of  the  appraisal  reference  model  (process  areas),  must  be  defined 
and  agreed  upon.  The  scope  of  the  organization  and  model  provides  the  basis  on  which  to  estimate 
personnel  time  commitments,  logistical  costs  (e.g.,  travel),  and  overall  costs  to  the  appraised 
organization  and  to  the  sponsoring  organization. 

During  the  appraisal,  the  appraisal  team  verifies  and  validates  the  objective  evidence  provided  by 
the  appraised  organization  to  identify  strengths  and  weaknesses  relative  to  the  appraisal  reference 
model.  Objective  evidence  consists  of  artifacts  and  affirmations  used  as  indicators  for 
implementation  and  institutionalization  of  model  practices.  Before  the  Conduct  Appraisal  phase 
begins,  members  of  the  appraised  organization  typically  collect  and  organize  documented 
objective  evidence,  using  defined  data  collection  strategies  based  on  the  extent  of  artifacts 
available  within  the  organization  and  aligned  with  the  appraisal  reference  model. 

Advance  preparation  by  both  the  appraisal  team  and  the  appraised  organization  is  key  to  the  most 
efficient  execution  of  the  method.  Analysis  of  preliminary  documented  objective  evidence 
provided  by  the  appraised  organization  plays  an  important  role  in  preparing  for  appraisal 
execution.  If  substantial  data  are  missing  at  this  point,  subsequent  appraisal  activities  can  be 
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delayed  or  even  cancelled  if  the  judgment  is  made  that  continuing  appraisal  activities  will  not  be 
sufficient  to  make  up  for  the  deficiency  given  the  resources  available. 

The  collection  of  documented  objective  evidence,  to  some  degree,  by  the  appraised  organization 
in  advance  of  the  appraisal  can  help  improve  appraisal  team  efficiency,  but  can  also  offer  several 
other  benefits  to  the  organization: 

•  improved  accuracy  in  appraisal  results  delivered  by  external  appraisal  teams  (i.e.,  clear 
understanding  of  implemented  processes,  strengths,  and  weaknesses) 

•  obtaining  a  detailed  understanding  of  how  each  part  of  the  organization  participating  in  the 
appraisal  has  implemented  model  practices,  and  the  degree  of  compliance  and  tailoring  of 
organizational  standard  processes 

•  establishing  residual  appraisal  assets  that  can  be  reused  on  subsequent  appraisals,  minimizing 
the  effort  necessary  for  preparation 

However,  the  effort  to  collect,  organize,  and  review  large  amounts  of  objective  evidence  in 
advance  of  the  appraisal  can  be  a  large  cost  to  appraised  organizations,  and  can  lead  to 
diminishing  returns  if  not  done  efficiently.  Incremental  data  collection  strategies  with  specific 
data  requests  can  help  mitigate  the  risks  of  inefficiently  using  the  organization’s  resources  on 
collecting  data  that  is  not  appropriate  or  useful.  A  data  collection  plan,  developed  by  the  appraisal 
team  leader  in  conjunction  with  the  appraised  organization,  can  help  make  explicit  the  choices  on 
how  much  data  collection  effort  to  distribute  between  the  organization  and  the  appraisal  team. 

Phase  2:  Conduct  Appraisal 

In  phase  2,  the  appraisal  team  focuses  on  collecting  data  from  the  appraised  organization  to  judge 
the  extent  to  which  the  model  is  implemented.  Integral  to  this  approach  is  the  concept  of 
coverage,  which  implies  two  things:  first,  the  collection  of  sufficient  data  for  each  model 
component  within  the  model  scope  selected  by  the  sponsor,  and  second,  obtaining  a  representative 
sample  of  ongoing  processes. 

Sampling  is  planned  quantitatively  based  on  the  diversity  of  unique  process  implementations 
within  the  appraisal  scope,  with  the  goal  of  both  ensuring  a  representative  sample  of  the 
organizational  unit  and  optimizing  the  effort  for  collection  and  analysis  of  objective  evidence. 

This  means  collecting  data  and  information  on  all  the  appraisal  reference  model  practices  in  the 
appraisal  scope,  across  sampled  process  instantiations  within  the  organizational  unit  being 
appraised.  The  data  collection  plan  developed  in  phase  1  undergoes  continuous  iteration  and 
refinement  until  sufficient  coverage  is  achieved. 

Upon  determining  that  sufficient  coverage  of  the  appraisal  reference  model  and  organizational 
unit  has  been  obtained,  appraisal  findings  and  ratings  may  be  generated.  Goal  ratings  are 
detennined  within  each  process  area,  which  collectively  can  be  used  to  detennine  aggregate 
ratings  for  the  individual  process  areas  or  for  the  organizational  unit,  as  appropriate. 

Phase  3:  Report  Results 

In  phase  3,  the  appraisal  team  provides  the  findings  and  ratings  to  the  appraisal  sponsor  and  the 
organization.  These  artifacts  become  part  of  the  appraisal  record,  which  becomes  protected  data  in 
accordance  with  the  appraisal  disclosure  statement.  The  level  of  protection  and  the  plan  for  the 
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disposition  of  appraisal  materials  and  data  are  determined  in  phase  1  in  collaboration  with  the 
sponsor.  A  completed  appraisal  data  package,  which  includes  a  subset  of  the  contents  of  the 
appraisal  record,  is  provided  to  the  SEI.  The  SEI  adds  the  appraisal  data  to  confidential  databases, 
and  provides  overall  profiles  of  the  community  on  a  periodic  basis. 

SCAMPI  A  Tailoring 

Successful  application  of  SCAMPI  A  relies  on  adjusting  the  parameters  of  the  method  to  the 
needs  of  the  organization  and  to  the  objectives  and  constraints  of  the  sponsor’s  organization. 

The  sponsor’s  objectives  largely  influence  tailoring  decisions.  The  reference  model  scope  and 
representation  (staged  or  continuous),  the  size  of  the  organizational  unit,  the  parts  of  the 
organization  sampled,  the  size  of  the  appraisal  team,  and  the  number  of  interviews  greatly 
influence  things  such  as  preparation  time,  time  on  site,  and  monetary  costs,  and  so  are  also  major 
factors  when  choosing  tailoring  options.  All  tailoring  decisions  must  be  documented  in  the 
appraisal  plan. 

Tailoring  provides  flexibility  to  efficiently  adapt  the  appraisal  to  the  needs  of  the  appraisal 
sponsor,  within  acceptable  limits  allowed  by  the  appraisal  method.  The  appraisal  team  leader  is 
responsible  for  ensuring  that  the  requirements  of  the  method  are  satisfied.  Tailoring  the  method 
too  severely  could  result  in  failure  to  satisfy  method  requirements,  the  inability  to  obtain  sufficient 
data  for  generation  of  appraisal  findings  or  ratings,  or  failure  to  meet  the  criteria  necessary  for 
recognition  as  a  valid  SCAMPI  A  appraisal. 

Time  Frame  and  Personnel  Requirements 

A  requirement  of  the  SCAMPI  A  method  is  that  the  Conduct  Appraisal  phase  must  be  completed 
within  90  days.  Afterwards,  the  follow-on  activities  implicit  with  a  full  cycle  of  appraisal  to 
reappraisal  would  typically  include  an  extended  period  of  time  for  creating  an  action  plan  and 
implementing  improved  processes,  with  a  reappraisal  occurring  in  the  latter  six  months  of  that 
period.  SCAMPI  A  appraisal  results  are  valid  for  a  timeframe  of  up  to  36  months. 

Personnel  needed  to  participate  in  activities  or  perform  tasks  in  a  SCAMPI  A  appraisal  include  the 
sponsor,  the  appraisal  team  leader,  the  appraisal  coordinator,  selected  participants,  and  appraisal 
team  members.  Their  time  commitments  will  vary  widely  depending  on  the  specific  parameters  of 
the  appraisal  (e.g.,  organizational  scope)  and  their  role. 

Much  of  the  effort  spent  by  an  organization  preparing  for  an  appraisal  is  for  the  collection  and 
review  of  objective  evidence;  these  costs  can  vary  widely  based  on  the  data  collection  strategy 
and  acceptable  level  of  risk.  Excluding  data  collection  costs,  appraisal  participants  can  typically 
expect  to  spend  one  to  three  hours  each  to  participate  in  interviews  and  attend  validation  sessions, 
plus  one  to  three  hours  each  for  presentations.  On  the  other  extreme,  the  appraisal  coordinator 
may  spend  many  weeks  of  full-time  effort  helping  the  team  and  the  organization  to  prepare  for 
and  conduct  the  appraisal.  Appraisal  team  leaders  should  engage  appraisal  sponsors  on  effort 
estimates  and  the  set  of  tailoring  options  to  be  used  in  conducting  a  SCAMPI  A  appraisal. 
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SCAMPI  A  Method  Overview 


This  section  provides  an  overview  of  the  underlying  principles  and  concepts  of  the  SCAMPI  A 
method.  Readers  of  the  SCAMPI  A  MDD  should  become  familiar  with  this  material  prior  to 
reading  the  process  descriptions  in  Part  II,  where  the  method  requirements  and  tailoring  options 
are  defined.  This  overview  is  primarily  targeted  at  appraisal  team  leaders  and  appraisal  team 
members  who  will  be  performing  SCAMPI  A  appraisals.  Additional  audiences  might  include 
appraisal  sponsors  or  process  improvement  professionals  interested  in  understanding  SCAMPI  A 
features  and  the  results  that  can  be  expected. 

Method  Context 

The  SCAMPI  A  appraisal  method  is  used  to  identify  strengths,  weaknesses,  and  ratings  relative  to 
appraisal  reference  models.  It  incorporates  best  practices  recognized  as  successful  in  the  appraisal 
community,  and  is  based  on  the  features  of  several  legacy  appraisal  methods. 

SCAMPI  A  satisfies  the  Appraisal  Requirements  for  CMMI  (ARC)  VI. 3  and  is  a  Class  A 
appraisal  method. 

Method  Objectives  and  Characteristics 

The  SCAMPI  A  method  has  the  following  primary  objectives: 

•  provide  a  common,  integrated  appraisal  method  capable  of  supporting  appraisals  in  the 
context  of  internal  process  improvement,  supplier  selection,  and  process  monitoring  (see 
“Modes  of  Usage”  on  page  18) 

•  provide  an  efficient  appraisal  method  capable  of  being  implemented  within  reasonable 
performance  constraints  (see  “Conducting  Cost-Effective  Appraisals”  on  page  28) 

The  SCAMPI  A  method  is  also  designed  to  prioritize  and  satisfy  certain  essential  characteristics, 
which  were  obtained  via  community  feedback  and  are  summarized  in  Table  6.  These 
characteristics  have  been  used  as  the  rationale  for  key  method  architecture  and  design  decisions, 
which  are  described  in  this  overview  and  throughout  the  MDD. 
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Table  6:  Essential  Characteristics  of  the  SCAMPI  A  Method 


Characteristic 

Explanation 

Accuracy 

Appraisal  ratings  truly  reflect  the  organization’s  maturity/capability,  reflect  the 
appraisal  reference  model,  and  can  be  used  for  comparison  across 
organizations. 

Appraisal  results  reflect  the  strengths  and  weaknesses  of  the  appraised 
organization  (i.e.,  no  exemplary  strengths  and  weaknesses  are  left 
undiscovered). 

Repeatability 

The  ratings  and  findings  of  an  appraisal  are  likely  to  be  consistent  with  those  of 
another  independent  appraisal  conducted  under  comparable  conditions  (i.e., 
another  appraisal  of  identical  scope  will  produce  consistent  results). 

Cost/Resource 

Effectiveness 

The  appraisal  method  is  efficient  in  terms  of  person-hours  spent  planning, 
preparing,  and  executing  an  appraisal. 

The  method  takes  account  of  the  organizational  investment  in  obtaining  the 
appraisal  results,  including  the  resources  of  the  host  organization,  the  impact  on 
the  appraised  organization,  and  the  appraisal  team. 

Meaningfulness 
of  Results 

Appraisal  results  are  useful  to  the  appraisal  sponsor  in  supporting  decision 
making.  This  support  of  decision  making  may  include  application  of  the 
appraisal  results  in  the  context  of  internal  process  improvement,  supplier 
selection,  or  process  monitoring. 

ARC  Compliance 

SCAMPI  A  is  a  Class  A  method  and  complies  with  all  ARC  requirements. 

Summary  of  SCAMPI  VI. 3  Changes 

Updates  to  the  SCAMPI  A  method  for  VI. 3  are  based  on  community  change  requests,  grouped 
into  three  overarching  goals  and  improvement  objectives,  as  summarized  in  Table  7.  SCAMPI 
VI. 3  method  features  that  implement  these  goals  and  objectives  are  further  described  in  the 
remainder  of  the  MDD  and  detailed  in  Part  II  process  descriptions. 
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Table  7;  Summary  of  SCAMPI  A  VI. 3  Updates 


SCAMPI  VI. 3 
Goals 

SCAMPI  VI .3 
Objectives 

SCAMPI  Essential 
Characteristics 

SCAMPI  VI. 3  Change  Concepts 

1.  Increase 
Efficiency  of 
the  Appraisal 
Process 

•  Consider  entire 
lifecycle  of 
cost/value  (not 
just  ‘onsite’) 

•  Decrease  cost 
while  maintaining 
accuracy  and 
utility 

•  Increase  value 
returned  per  cost 
incurred 

•  Cost/Resource 
Effectiveness 

•  Accuracy 
Meaningfulness 
of 

Results 

•  Appraisal  scoping:  analysis  of 
sampling  factors  to  assure 
representative  coverage  and 
efficient  data  collection.  (Replaces 
focus/non-focus  projects.) 

•  Clarify  handling  of  basic  units  and 
support  functions  (sufficiency, 
coverage) 

•  Formalized  plans  for  data 
collection  and  resources 

•  Managed  discovery,  iterative  data 
calls  (limit  PHD  crafting) 

•  Removed  distinction  of  direct  and 
indirect  artifacts 

•  Multi-constellation  appraisals 

2.  Remove 
Barriers  to 
Broader 

Usage  (e.g., 
other 

constellations) 

•  Remove 
terminology 
unique  to  CMMI- 
DEV 

•  Clarify 
skill/experience 
requirements  for 
all  users 

•  Meaningfulness 
of 

Results 

•  Repeatability 

•  Terminology: 

-organization:  “project”  >  “basic 
unit” 

-term  “indirect  artifact”  not  used 
-sampling  “subgroups” 

•  Appraisal  team  qualifications  and 
field  experience  for  model  scope; 
manage  conflicts  of  interest 

•  Multi-constellation  appraisals 

3.  Synchronize 
with  CMMI 

VI. 3  Product 
Suite 

•  Enhance 
consistency  of 
usage  and  fidelity 
to  method 
requirements 

•  Evolve  methods 
based  on  change 
requests 

•  Repeatability 

•  ARC  Compliance 

•  Consistency  with  other  VI. 3 
product  suite  (ARC/model) 
updates 

•  MDD  clarifications,  cleanup,  errata 

•  Simplified  features/terms  from 
removal  of  ARC  ISO/I  EC  15504 
conformance  (e.g.,  appraisal 
input). 
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Key  SCAMPI  VI. 3  MDD  Change  Concepts 


Broadened  Applicability  of  SCAMPI  Method 

A  primary  objective  for  SCAMPI  VI. 3,  as  depicted  in  Table  7,  is  to  synchronize  with  other 
components  of  the  CMMI  product  suite.  Though  SCAMPI  has  been  applied  in  many  contexts,  the 
MDD  legacy  was  primarily  written  to  support  single-model  CMMI-DEV  appraisals.  This 
sometimes  made  it  difficult  to  apply  the  method  to  other  reference  models,  such  as  CMMI  for 
Services  (CMMI-SVC)  or  People  CMM  (P-CMM).  The  SCAMPI  VI. 3  MDD  uses  more  generic 
terminology  to  better  accommodate  the  architectures  used  in  these  other  reference  models.  In 
addition,  guidance  has  been  added  (refer  to  Appendix  G)  on  how  to  conduct  a  SCAMPI  appraisal 
using  more  than  one  reference  model  (i.e.,  multi-model  appraisals). 

Terminology  Changes 

Several  different  terms  are  used  or  emphasized  in  the  SCAMPI  VI. 3  MDD,  largely  due  to  the 
broadened  applicability  (described  above)  or  particular  focus  areas  for  method  improvements. 
(Reference  the  Glossary  to  read  definitions.)  This  does  not  imply  these  prior  concepts  from 
SCAMPI  VI. 2  no  longer  exist,  but  rather  that  they  are  reflected  differently  due  to  emphasis  in 
SCAMPI  VI  .3.  Some  of  the  key  terminology  changes  evident  in  the  SCAMPI  V 1 .3  MDD 
include: 

•  Basic  units:  used  throughout  the  document  to  identify  blocks  of  work  or  people  who  form  the 
elements  of  the  organizational  unit  to  be  sampled.  Previously,  this  would  have  been  called 
“project”  in  the  context  of  CMMI-DEV.  The  concept  of  “support  function”  has  also  been 
introduced  to  explicitly  account  for  other  structures  within  the  organizational  unit  that  do  not 
tend  to  do  “customer-facing  work.”  Examples  of  support  functions  include  Quality 
Assurance  or  Process  Engineering  groups.  Organizations  will  define  basic  units  and  support 
functions  to  suit  their  needs  -  it  is  not  presumed  that  these  terms  will  imply  the  same  things  in 
every  organizational  unit. 

•  Artifacts:  the  distinction  between  direct  and  indirect  artifacts  has  been  removed  in  SCAMPI 
VI. 3.  This  distinction  was  often  confusing  and  could  unnecessarily  drive  data  collection  costs 
higher.  The  more  general  term  artifact  is  used  now. 

•  Practice  Implementation  Indicator  (PII):  PIIs,  and  the  related  term  Practice 
Implementation  Indicator  Description  (PHD  ),  were  used  in  prior  MDD  versions  primarily  to 
describe  internal  appraisal  team  data  structures  for  organizing  objective  evidence.  Their 
optional  use  by  appraised  organizations  to  collect  artifacts  in  a  verification-based  appraisal 
was,  however,  over-interpreted  in  practice  and  greatly  influenced  ineffective  and  costly  data 
collection.  To  break  some  of  these  ineffective  practices,  these  tenns  have  been  replaced  in 
VI. 3  with  the  labels  objective  evidence  and  database  of  objective  evidence.  See  the  Data 
Collection  section  for  further  discussion. 
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•  Focus  and  non-focus  projects:  this  V 1 .2  concept  is  reflected  in  VI  .3  data  sampling  and 
coverage  rules,  which  still  require  some  number  of  basic  units  to  provide  objective  evidence 
for  most,  if  not  all  process  areas,  while  some  basic  units  provide  evidence  for  fewer  process 
areas.  The  terms  ‘focus  project’  and  ‘non-focus  project’  were  removed  in  VI. 3  to  better 
reflect  the  new  sampling  approach  and  to  avoid  confusion  that  has  occasionally  occurred  with 
these  terms  in  prior  MDD  versions. 

•  Corroboration:  the  concept  of  corroboration  of  objective  evidence,  required  by  the  ARC  and 
fundamental  to  verifying  accurate  data  collection  for  appraisals,  is  reflected  in  revised  data 
coverage  rules  for  VI. 3.  The  concept  is  designed  directly  into  the  method  rules  even  though 
the  term  ‘corroboration’  itself  is  de-emphasized. 

•  Appraisal  input:  due  to  the  removal  of  the  ARC  requirement  for  1SO/1EC  15504 
conformance,  this  term  is  no  longer  used  in  SCAMPI  VI  .3.  Content  previously  required  by 
the  appraisal  input  is  reflected  in  the  initial  appraisal  plan,  consistent  with  current  common 
practice. 

Appraisal  Scoping  and  Sampling 

Planning  and  selection  of  the  organizational  scope  of  the  appraisal  is  now  based  on  quantitative 
sampling  rules  to  ensure  adequate  representation  of  basic  units  across  the  organizational  unit, 
while  also  offering  opportunities  to  optimize  data  collection  and  appraisal  costs.  Selection  is  based 
on  analysis  of  sampling  factors  (formerly  called  critical  factors  in  V 1 .2)  and  subgroups  that 
characterize  differing  definitions  or  implementations  of  the  processes  in  use  within  the 
organizational  unit.  Details  on  appraisal  scoping  and  sampling,  including  the  fonnula  required  to 
ensure  representative  coverage  of  basic  units  within  sampling  subgroups,  are  further  described  in 
activity  1.1.4,  Determine  Appraisal  Scope  and  Appendix  F,  Scoping  and  Sampling  in  SCAMPI  A 
Appraisals. 

Data  Coverage  Rules 

(To  read  more,  refer  to  activity  1.1.4,  Determine  Appraisal  Scope) 

With  the  change  to  a  sampling-based  approach  in  SCAMPI  VI. 3,  new  method  rules  are  defined 
for  the  amount  of  objective  evidence  (artifacts  and/or  affirmations)  that  must  be  provided  for 
basic  units  and  support  functions  within  each  sampling  subgroup.  These  data  coverage  mles, 
which  vary  for  basic  units  and  support  functions,  also  define  minimum  constraints  on  process 
areas  for  which  objective  evidence  must  be  provided,  based  on  the  quantity  of  samples  within  the 
sampling  subgroup  and  differences  of  process  implementation.  Data  coverage  rules  in  V 1 .3  are 
conceptually  similar  to  SCAMPI  VI. 2  method  rules  for  corroboration  and  the  extent  to  which 
verbal  affirmations  were  required  across  process  instantiations,  but  redefined  to  ensure  adequate 
coverage  and  confidence  in  results  obtained  using  the  V 1 .3  sampling  approach. 
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Data  Collection 

Much  of  the  organizational  cost  incurred  for  conducting  SCAMPI  A  appraisals  has  been  attributed 
to  the  inefficient  collection  of  data  provided  in  (optional)  PIID  data  structures  for  verification- 
based  appraisals,  resulting  in  much  effort  lost  by  both  the  organization  and  the  appraisal  team  (see 
the  topic  Conducting  Cost-Effective  Appraisals  for  further  discussion).  Several  SCAMPI  VI. 3 
method  features  have  been  improved  to  make  data  collection  more  efficient  and  cost-effective, 
including  the  following: 

•  De-emphasizing  PIID  data  structures  and  distinctions  between  direct  and  indirect  artifacts 

•  Sampling  approaches  to  obtain  better  representation  with  potentially  less  data  collection 
impact 

•  Defining  the  concept  of  managed  discovery,  balancing  verification-based  and  discovery- 
based  techniques  with  iterative  data  calls  to  close  specific  gaps  in  objective  evidence  needed. 
Guidance  is  also  provided  for  product-based  data  collection  with  one-to-many  mapping  to 
reference  model  practices  (refer  to  Appendix  E,  Managed  Discovery  Collection). 

•  Adding  emphasis  on  defining  and  maintaining  a  data  collection  strategy,  negotiated  with  the 
appraisal  sponsor  as  part  of  the  appraisal  plan  (refer  to  1.1.2,  Determine  Data  Collection 
Strategy  and  1.2.3,  Develop  Data  Collection  Plan) 

These  improvements  are  expected  to  optimize  appraisal  data  collection  (“just  enough  data  and  no 
more”)  with  a  better  balance  of  effort  between  the  appraisal  team  and  the  appraised  organization. 

Appraisal  Team  Qualifications 

Method  requirements  for  appraisal  team  selection  have  been  clarified  for  team  member  training, 
qualifications,  and  experience  in  the  domains  addressed  by  the  reference  models  used.  Objectivity 
is  assured  by  mitigating  potential  conflicts  of  interest  (refer  to  1 .3.2,  Select  Team  Members  and 
1.3.3,  Document  and  Manage  Conflicts  of  Interest). 

Modes  of  Usage 

As  an  ARC  Class  A  method,  SCAMPI  A  is  primarily  a  method  oriented  towards  benchmarking 
and  suitable  for  generating  ratings.  SCAMPI  A  appraisals  can  be  performed  in  three  modes  of 
usage,  as  depicted  in  Table  8. 
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Table  8:  SCAMPI  A  Modes  of  Usage 


Usage  Mode 

Description 

Internal  Process 
Improvement 

Organizations  use  appraisals  to  appraise  internal  processes,  generally  to  either 
baseline  their  capability/maturity  level(s),  to  establish  or  update  a  process 
improvement  program,  or  to  measure  progress  in  implementing  such  a  program. 
Applications  include  measuring  process  improvement  progress,  conducting 
process  audits,  focusing  on  specific  domains  or  product  lines,  appraising  specific 
parts  of  the  organization,  and  preparing  for  external  customer-led  appraisals.  In 
this  manner,  SCAMPI  A  appraisals  supplement  other  tools  for  implementing 
process  improvement  activities. 

Supplier  Selection 

Appraisal  results  are  used  as  a  high-value  discriminator  to  select  suppliers.  The 
results  are  used  in  characterizing  the  process-related  risk  of  awarding  a  contract 
to  a  supplier.  The  appraisal  results  are  typically  only  one  criterion  among  many 
used  to  select  suppliers.  Results  are  often  used  as  a  baseline  in  subsequent 
process  monitoring  with  the  selected  supplier. 

Process  Monitoring 

Appraisal  methods  are  also  used  in  monitoring  processes  (e.g.,  after  contract 
award,  by  serving  as  input  for  an  incentive/award  fee  decision  or  a  risk 
management  plan).  The  appraisal  results  are  used  to  help  the  sponsoring 
organization  tailor  its  contract  or  process  monitoring  efforts  by  allowing  it  to 
prioritize  efforts  based  on  the  observed  strengths  and  weaknesses  of  the 
supplying  organization’s  processes.  This  usage  mode  focuses  on  a  long-term 
teaming  relationship  between  the  sponsoring  organization  and  the  development 
organization  (i.e.,  buyer  and  supplier). 

Where  appropriate,  differences  in  the  method  requirements,  tailoring,  or  recommended 
implementation  applicable  to  these  usage  modes  are  discussed  in  process  descriptions  and 
activities  provided  in  Part  II.  These  differences  occur  most  significantly  in  the  planning  processes 
(e.g.,  appraisal  objectives,  sponsorship,  appraisal  planning,  selection  of  participants,  and 
preparation)  and  reporting  processes  (e.g.,  reporting  of  appraisal  results,  use  of  appraisal  results 
for  decision  making,  and  follow-on  activities). 

With  the  extension  of  the  SCAMPI  method  family  to  include  ARC  Class  B  and  Class  C  methods, 
appraisal  sponsors  for  these  usage  modes  are  encouraged  to  consider  the  SCAMPI  B  or  SCAMPI 
C  appraisal  methods  which  may  produce  acceptable  results  where  ratings  are  not  required,  with 
significantly  less  investment  of  cost  and  resources.  Refer  to  “Requirements  for  CMMI  Appraisal 
Method  Class  Structure”  and  “Requirements  for  CMMI  Appraisal  Methods”  in  the  ARC  for 
further  discussion  of  these  issues  and  for  guidance  in  selecting  an  appropriate  appraisal  method  to 
fit  desired  business  objectives. 
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Method  Concepts 

This  section  describes  fundamental  concepts  employed  by  the  SCAMPI  A  method.  These 
concepts  are  treated  here  to  provide  readers  with  an  overall  understanding  of  the  method  prior  to 
reading  the  method  Process  Definitions  in  Part  II.  Many  of  these  concepts  are  distributed  across 
several  appraisal  method  processes  or  activities,  so  it  is  important  to  ensure  that  a  common 
understanding  is  obtained  to  recognize  the  components  of  these  concepts  as  they  appear  elsewhere 
in  this  document. 

In  addition  to  requirements  of  the  ARC,  these  method  concepts  are  derived  from,  and  heavily 
influenced  by,  the  method  objectives  and  essential  method  characteristics. 

Method  Assumptions  and  Design  Principles 

In  addition  to  method  objectives  and  characteristics,  SCAMPI  A  features  are  based  on  certain 
method  assumptions  and  design  principles,  described  below,  related  to  the  expected  use  of  the 
method. 

SCAMPI  A  is  a  Class  A  benchmarking  method. 

As  an  ARC  Class  A  method,  SCAMPI  A  can  be  used  to  generate  ratings  as  benchmarks  to 
compare  maturity  levels  or  capability  levels  across  organizations.  As  a  benchmarking  method,  the 
SCAMPI  A  emphasis  is  on  a  rigorous  method  capable  of  achieving  high  accuracy  and  reliability 
of  appraisal  results  through  the  collection  of  objective  evidence  from  multiple  sources. 

Goal  ratings  are  a  function  of  the  extent  to  which  the  corresponding  reference  model  practices 
are  present  in  the  planned  and  implemented  processes  of  the  organization. 

In  the  CMMI  appraisal  reference  models,  there  is  a  direct  relationship  between  goals  and  the 
practices  that  contribute  toward  achievement  of  those  goals.  Goals  are  required  model 
components;  practices  are  expected  model  components  in  that  alternative  practices  could  be 
implemented  that  are  equally  effective  in  achieving  the  intent  of  the  associated  goals. 

In  the  SCAMPI  A  method,  a  fundamental  premise  is  that  satisfaction  of  goals  can  be  determined 
only  upon  detailed  investigation  of  the  extent  to  which  each  corresponding  practice  is 
implemented  for  each  sample  instantiation  used  as  a  basis  for  the  appraisal  (i.e.,  basic  units  and 
support  functions). 

Additional  information  on  rating  goals  is  provided  in  “Data  Collection,  Rating,  and  Reporting”  in 
this  document. 

The  aggregate  of  objective  evidence  provided  is  used  as  the  basis  for  determining  practice 
implementation. 

To  make  reasonable  judgments  regarding  an  organization’s  implemented  processes  relative  to  the 
appraisal  reference  model,  appraisal  teams  base  their  judgments  on  the  collection  of  objective 
evidence  for  each  practice  applicable  to  process  area  goals  within  the  appraisal  reference  model 
scope. 
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Appraisal  teams  compare  the  objective  evidence  collected  against  the  corresponding  practices  in 
the  appraisal  reference  model.  In  making  inferences  about  the  extent  to  which  practices  are  or  are 
not  implemented,  appraisal  teams  draw  on  the  entire  model  document  to  understand  the  intent  of 
the  model,  and  use  it  as  the  basis  for  their  decisions.  This  comparison  includes  the  required  and 
expected  model  components  (i.e.,  goals  and  practices)  as  well  as  informative  material,  such  as 
model  front  matter,  introductory  text,  glossary  definitions,  and  subpractices. 

Practice  implementation  at  the  organizational  unit  level  is  a  function  of  the  degree  of practice 
implementation  at  the  instantiation  level. 

Practices  described  in  the  CMMI  appraisal  reference  models  are  abstractions  that  are  realized  by 
their  implementation  within  organizations,  and  instantiated  at  the  level  of  basic  units  and  support 
functions  in  the  organizational  unit.  The  context  within  which  the  practice  is  applied  drives  the 
implementation.  The  details  of  the  implementation,  as  well  as  the  context  within  which  the 
practice  is  implemented,  are  referred  to  as  the  instantiation  of  the  practice. 

An  organizational  unit  is  the  part  of  an  organization  that  is  the  focus  of  an  appraisal.  An 
organizational  unit  operates  within  a  coherent  process  context  and  a  coherent  set  of  business 
objectives.  (Refer  to  the  Glossary  for  a  complete  definition  of  organizational  unit.) 

The  extent  to  which  an  organizational  unit  has  implemented  appraisal  reference  model  practices 
can  be  determined  only  by  considering,  in  aggregate,  the  extent  to  which  those  practices  are 
implemented  within  the  organizational  unit  by  basic  units  and  support  functions.  This  process,  in 
turn,  necessitates  the  consideration  of  objective  evidence  for  each  instantiation,  for  each  model 
practice  within  the  appraisal  reference  model  scope. 

Appraisal  teams  are  obligated  to  seek  and  consider  objective  evidence  of  multiple  types  in 
determining  practice  implementation  and  goal  satisfaction. 

The  SCAMPI  A  method  is  data-oriented  in  that  decisions  on  practice  implementation  and  goal 
rating  are  made  based  on  the  aggregate  of  objective  evidence  available  to  the  appraisal  team. 
Multiple  types  of  objective  evidence  (artifacts  and  affirmations)  must  be  considered;  these  types 
are  described  in  Types  of  Objective  Evidence  in  this  document.  Artifacts  indicative  of  practice 
implementation  are  a  requirement  of  the  SCAMPI  A  method. 

Affirmations  are  required  to  ensure  that  the  documentation  reflects  the  actual  organizational 
process  implementation  and  to  preclude  rating  judgments  being  made  solely  on  the  basis  of 
artifacts.  The  SCAMPI  A  method  establishes  minimum  requirements,  called  data  coverage  rules 
(described  in  activity  1.1.4,  Determine  Appraisal  Scope)  for  the  extent  to  which  objective 
evidence  from  affirmations  must  be  collected  from  basic  units  and  support  functions  for  model 
practices  to  corroborate  other  sources  of  objective  evidence  prior  to  rating  goals. 

Types  of  Objective  Evidence 

The  fundamental  idea  behind  SCAMPI  appraisals  is  that  the  conduct  of  an  activity  or  process 
results  in  footprints  or  objective  evidence,  that  substantiates  work  being  done  consistent  with 
appraisal  reference  model  practices.  For  example,  the  establishment  of  an  artifact,  such  as  a 
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document,  is  often  an  expected  outcome  resulting  from  implementation  of  a  model  practice.  Other 
artifacts  may  also  substantiate  implementation  of  the  practice,  such  as  evidence  of  a  status 
meeting  or  review  being  held.  Members  of  the  organizational  unit  may  affirm  through  interviews 
how  practices  are  implemented. 

The  SCAMPI  A  method  provides  for  the  collection  and  analysis  of  data  from  the  following  types 
of  objective  evidence: 

•  artifacts  -  a  tangible  form  of  objective  evidence  indicative  of  work  being  performed  that 
represents  either  the  primary  output  of  a  model  practice  or  a  consequence  of  implementing  a 
model  practice.  These  artifacts  may  include  organizational  policies,  meeting  minutes,  review 
results,  or  other  implementation-level  work  products.  Sufficient  artifacts  demonstrating  and 
corroborating  that  the  work  is  being  done  are  necessary  to  verify  the  implementation  of 
associated  model  practices. 

•  affirmations  -  an  oral  or  written  statement  confirming  or  supporting  implementation  (or  lack 
of  implementation)  of  a  model  practice  provided  by  the  implementers  of  the  practice, 
provided  via  an  interactive  forum  in  which  the  appraisal  team  has  control  over  the  interaction. 
These  statements  are  typically  collected  using  interviews,  demonstrations,  questionnaires,  or 
other  means.  Note  that  negative  affirmations  confirming  the  lack  of  implementation  of  a 
practice  are  possible. 

Using  multiple  data-gathering  mechanisms  improves  the  team’s  depth  of  understanding  and 
enables  corroboration  of  the  data.  An  over-reliance  on  one  type  of  objective  evidence  or  another  is 
undesirable.  Too  much  dependence  on  artifacts  could  result  in  the  perception  that  the  appraisal 
was  a  “paper  review”  (i.e.  a  cursory  examination  of  prepared  documents)  and  not  truly  indicative 
of  organizational  and/or  project  behavior.  An  over-reliance  on  affirmations  could  be  criticized  as 
not  truly  objective  or  repeatable.  Therefore,  the  SCAMPI  A  method  requires  a  balance  across 
these  types  of  objective  evidence,  described  in  data  coverage  rules  (refer  to  1.1.4,  Determine 
Appraisal  Scope). 

The  extent  to  which  objective  evidence  is  judged  appropriate  to  determine  a  given  practice  is 
implemented  (or  not  implemented)  will  vary  according  to  the  context  in  which  the  process  is 
implemented,  and  influenced  by  factors  such  as  size,  organizational  culture,  application  domain, 
customer  market,  and  so  on.  For  example,  the  level  of  detail  necessary  for  a  work  breakdown 
structure  will  differ  widely  for  a  one-person,  two-week  maintenance  effort  as  opposed  to  a  1 00- 
person,  multi-year,  and  mission-critical  development  of  a  new  product. 

Data  Collection  Approaches 

SCAMPI  A  appraisals  are  data-intensive  activities,  involving  the  collection,  organization,  and 
evaluation  of  a  large  set  of  artifacts  that  substantiate  the  implementation  of  appraisal  reference 
model  practices.  Data  collection  is  typically  the  single  largest  driver  in  the  overall  cost  and 
schedule  for  conducting  SCAMPI  A  appraisals,  and  can  be  a  substantial  impact  on  resources 
within  the  appraised  organization.  Often  this  may  depend  on  the  extent  to  which  the  appraised 
organization  has  existing  assets  already  collected  and  in  place  to  serve  as  objective  evidence 
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provided  to  the  appraisal  team.  This  range  of  advance  data  readiness  can  be  characterized  by  the 
following  three  appraisal  data  collection  approaches: 

1.  Verification  approach:  For  organizations  that  already  have  detailed  collections  of  artifacts 
organized  by  mappings  to  practices  in  the  reference  model,  much  of  the  appraisal  team  effort 
can  concentrate  on  verifying  the  appropriateness  of  the  data  collected.  This  approach 
leverages  prior  investments  in  data  collection  by  the  organization  for  its  own  purposes  (such 
as  standardization,  process  compliance  monitoring,  or  product  reuse),  and  can  offer  great 
efficiencies  to  the  appraisal  team.  Rarely,  however,  has  this  ideal  been  fully  realized  in 
practice,  and  many  organizations  have  spent  huge  effort  creating  these  verification-ready 
appraisal  data  structures  essentially  from  scratch,  often  with  very  little  business  value  and 
frequently  resulting  in  much  rework  of  artifacts  that  are  inappropriate,  incomplete,  or 
overkill  for  what  is  needed.  Despite  the  potential  for  high  costs,  this  preparation-intensive 
approach  may  still  be  preferable  to  organizations  for  which  risk  of  achieving  maturity  level 
ratings  is  a  primary  concern. 

2.  Discovery  approach:  At  the  other  end  of  the  spectrum,  an  organization  may  have  few  such 
collections  of  artifacts  already  collected  and  mapped  to  the  reference  model.  The  appraisal 
team  must  then  work  directly  with  the  organization  to  identify,  locate,  organize,  and  evaluate 
these  artifacts  in  order  to  proceed  with  the  appraisal.  This  can  be  a  time-consuming  task  for 
the  appraisal  team,  and  may  involve  planning  for  higher  costs  and  a  longer  schedule  for  the 
additional  appraisal  team  effort,  however  it  limits  the  advance  preparation  needed  to  be  done 
by  the  organization.  This  approach  may  be  appropriate  for  some  situations,  such  as 
organizations  just  starting  a  process  improvement  initiative. 

3.  Managed  discovery:  An  incremental  (phased)  approach  to  data  collection  can  also  be  used. 
Initial  data  collection  is  focused  on  a  predetermined  set  of  high-leverage  work  products  that 
typically  provide  substantial  coverage  of  reference  model  practices  -  for  example,  plans, 
documents,  schedules,  measures,  and  reviews.  The  appraisal  team  maps  these  artifacts  to  the 
appraisal  reference  model,  and  determines  the  extent  to  which  gaps  in  model  coverage  still 
remain.  These  gaps  are  closed  by  iteratively  acquiring  more  data  through  a  series  of  specific 
data  calls.  This  approach  balances  the  distribution  of  effort  between  the  appraisal  team  and 
the  appraised  organization,  and  can  optimize  data  collection  by  focusing  on  the  work 
products  most  naturally  understood  by  the  organization  and  leaving  much  of  the  detailed 
model  knowledge  to  the  appraisal  team.  In  many  situations,  this  can  offer  the  most  cost- 
effective  approach  to  data  collection  and  best  use  of  an  organization’s  resources.  However, 
lack  of  an  initial  reference  model-centric  focus  could  also  obscure  the  potential  risk  an 
organization  may  face  for  model  compliance. 

Often  some  combination  of  these  approaches  may  be  used.  With  data  collection  such  a  large 
influence  on  the  cost  of  appraisals,  this  topic  is  given  much  emphasis  throughout  the  SCAMPI  A 
MDD,  notably  in  appraisal  planning  where  a  data  collection  strategy  (refer  to  1 . 1 .2,  Determine 
Data  Collection  Strategy)  and  data  collection  plan  (refer  to  1.2.3,  Develop  Data  Collection  Plan) 
must  be  negotiated  with  the  appraisal  sponsor  to  understand  the  choices  and  costs  involved. 
Appendix  E,  Managed  Discovery,  provides  more  guidance  on  the  managed  discovery  approach 
and  associated  tradeoffs. 
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Focused  Investigation 

In  order  to  achieve  efficient  appraisals,  emphasis  is  needed  not  only  on  effective  data  collection 
and  management  approaches,  but  also  on  the  methods  used  by  the  appraisal  team  to  maintain 
focus  on  the  end  objectives  and  to  prioritize  effort  where  it  most  needed.  The  term  focused 
investigation  is  used  to  describe  this  concept  of  optimized  investment  of  appraisal  team  resources, 
which  applies  throughout  the  SCAMPI  A  appraisal  processes.  Essentially,  this  approach  can  be 
described  at  a  high  level  using  the  following  paradigms: 

•  Understand  what  objective  evidence  is  available,  and  how  it  contributes  toward 
implementation  of  model  practices  within  the  appraisal  reference  model  scope. 

•  Continually  consolidate  data  to  determine  progress  toward  sufficient  coverage  of  model 
practices. 

•  Focus  appraisal  resources  by  targeting  those  areas  for  which  further  investigation  is 
necessary,  and  collect  additional  data  to  complete  the  set  of  objective  evidence  needed. 

•  If  sufficient  objective  evidence  has  been  gathered  to  conclude  a  practice  is  implemented,  there 
is  no  need  to  continue  working  on  it.  Stop  and  move  on  to  another  practice  where  adequate 
data  has  not  yet  been  gathered  or  where  the  appraisal  team  still  has  concerns. 

•  Avoid  unnecessary  or  duplicated  effort  that  does  not  contribute  additional  information  toward 
achievement  of  sufficient  coverage  or  toward  obtaining  significantly  greater  confidence  in  the 
appraisal  results.  For  example,  keep  interviews  efficient  by  asking  further  questions  only 
about  practices  for  which  sufficient  data  has  not  already  been  obtained. 

This  approach  begins  with  the  initial  collection  and  analysis  of  objective  evidence  from  the 
organizational  unit.  The  appraisal  team’s  inventory  of  objective  evidence  can  be  annotated  to 
identify  practices  that  are  strongly  supported,  or  those  that  need  further  clarification.  This 
knowledge  can  be  used  as  the  basis  for  determining  findings  that  affect  appraisal  outcomes. 

As  the  appraisal  process  progresses,  the  appraisal  team  aggregates  and  synthesizes  additional 
objective  evidence,  and  uses  this  evidence  to  draw  inferences  about  the  overall  implementation 
within  the  organizational  unit.  Wherever  there  are  shortcomings  in  the  appraisal  team’s 
understanding  of  the  organizational  unit’s  implementation  of  model  practices,  data  collection 
strategies  can  be  determined  to  probe  for  and  obtain  additional  information. 

For  example,  cases  where  the  objective  evidence  is  missing,  unclear,  or  insufficient  might  be 
addressed  through  additional  documentation  requests  or  by  generating  focused  questions  for 
specific  interview  participants.  By  maintaining  a  current  inventory  of  the  status  of  the  appraisal 
objective  evidence  and  prioritizing  areas  where  additional  information  is  still  needed,  these 
focused  investigation  approaches  can  be  continuously  and  iteratively  applied  to  narrow  remaining 
gaps  and  converge  on  sufficient  coverage  for  proceeding  with  rating. 
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Data  Collection,  Rating,  and  Reporting 

The  appraisal  team  follows  a  consensus-based,  structured  process  to  synthesize  and  transform 
information  collected  from  the  sources  described  in  “Types  of  Objective  Evidence”  in  this 
document.  Data  from  these  sources  are  collected  and  considered  in  several  discrete  data-gathering 
sessions,  either  as  integrated  appraisal  team  activities  or  by  subsets  of  the  team  organized  into 
mini-teams  operating  in  parallel.  Mini-teams  are  typically  organized  around  related  process  areas, 
with  mini-team  members  assigned  by  the  appraisal  team  leader  on  the  basis  of  their  individual 
experience,  knowledge,  and  skills. 


The  SCAMPI  A  data  transformation  and  rating  process  is  depicted  in  Figure  1. 


Level  of 
Consensus 


Capability  Level  and/or 
Maturity  Level  Ratings 


Full  Team 


Goal  Satisfaction  Ratings 


Full  Team 


Practice  Implementation  Characterizations 
(organizational  unit  level) 


Full  Team 


Practice  Implementation  Characterizations 
(practice  instantiation  level) 


Mini-Team 


Figure  1:  SCAMPI  A  Rating  Process 


Team  members  review  objective  evidence  provided  by  the  organizational  unit  and  determine  its 
relevance  to  the  appraisal  reference  model  practices. 

Areas  of  strength  or  weakness  observed  relative  to  the  implementation  of  appraisal  reference 
model  practices  are  recorded  in  written  findings.  Findings  are  generated  primarily  for  weaknesses 
or  gaps  of  the  implementation  compared  to  the  intent  of  a  model  practice. 

Findings  of  strengths  should  be  reserved  for  exemplary  practices  that  are  particularly  effective  and 
are  candidates  for  inclusion  in  aggregated  findings.  Observations  that  simply  reflect  a  sufficient 
implementation  of  a  practice  can  produce  substantial  data  management  overhead  that  does  not 
contribute  toward  generation  of  findings;  these  gratuitous  strengths  are  more  effectively  captured 
as  simple  indicators  in  the  appraisal  team’s  database  of  objective  evidence.  Findings  may  also  be 
generated  for  alternative  practices,  which  are  acceptable  alternatives  to  implementing  one  or  more 
model  practices  that  contribute  equivalently  to  the  satisfaction  of  process  area  goals. 
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Characterizing  Practices 

Verification  of  objective  evidence  continues  in  this  way  at  the  instantiation  level  (basic  units  and 
support  functions)  until  sufficient  objective  evidence  has  been  obtained  to  characterize  the 
implementation  of  an  appraisal  reference  model  practice.  Consensus  is  obtained  at  the  mini-team 
level  on  the  sufficiency  of  objective  evidence  and  the  accuracy  of  strengths  and  weaknesses. 

Based  on  the  objective  evidence  supporting  practice  implementation,  the  appraisal  team  assigns 
values  to  characterize  the  extent  to  which  the  appraisal  reference  model  practice  is  implemented. 

Each  practice  is  characterized  as  one  of  the  following  values: 

•  Fully  Implemented  (FI) 

•  Largely  Implemented  (LI) 

•  Partially  Implemented  (PI) 

•  Not  Implemented  (NI) 

.  Not  Yet  (NY) 

These  values  are  based  on  criteria  defined  by  the  method  (refer  to  2.4.2,  Characterize 
Implementation  of  Model  Practices  and  Generate  Preliminary  Findings). 

The  intent  of  this  characterization  is  to  summarize  the  appraisal  team’s  judgment  of  practice 
implementation  and  identify  and  prioritize  areas  where  further  judgment,  investigation,  or 
corroboration  may  be  necessary.  These  characterization  values  are  an  aid,  not  a  replacement,  for 
the  recorded  findings  of  weaknesses,  which  are  used  as  a  basis  for  rating  decisions. 

Upon  assigning  characterization  values  for  a  given  model  practice  for  each  basic  unit  or  support 
function,  the  characterization  values  are  aggregated,  using  full  appraisal  team  consensus,  to  the 
organizational  unit  level.  Weaknesses  across  the  basic  units  and  support  functions  are  similarly 
aggregated  to  the  organizational  unit  level,  and  form  the  basis  for  rating.  Where  team  judgment  is 
necessary  to  characterize  practice  implementation,  these  decisions  are  made  considering  factors 
such  as  the  mix  of  practice  characterizations,  the  reasons  for  the  supporting  instantiation-level 
characterizations,  and  the  severity  of  the  associated  weaknesses  (in  aggregate). 

Generating  Findings 

Strengths  and  weaknesses  identified  across  basic  units  and  support  functions  within  the 
organizational  unit  are  synthesized  and  aggregated  to  statements  of  preliminary  findings, 
expressed  at  the  organizational  unit  level.  These  strengths  and  weaknesses  are  often  organized  at 
the  level  of  process  area  goals  using  common  themes.  Preliminary  findings  are  provided  to  the 
organizational  unit  for  validation;  the  mechanisms  and  timeframe  used  for  this  validation  may 
vary  across  the  appraisal  modes  of  usage  (internal  process  improvement,  supplier  selection, 
process  monitoring). 
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During  this  activity,  the  appraisal  team  is  still  in  the  process  of  collecting  data  to  ensure  that  an 
accurate  understanding  of  the  organizational  process  implementation  is  obtained.  Feedback  from 
the  participants  in  the  appraisal  is  used  to  validate  the  preliminary  findings  and  may  result  in 
revised  or  additional  findings. 

The  appraisal  team  may  also  request  additional  data  sources  for  areas  where  their  understanding 
of  the  organizational  unit’s  implementation  of  model  practices  is  insufficient.  Final  findings  are 
generated  based  on  the  complete,  validated  set  of  appraisal  data  (i.e.,  findings  and  additional 
aggregated  strengths  and  weaknesses,  if  any). 

Generating  Ratings 

Ratings  are  generated  based  on  the  set  of  validated  appraisal  data.  At  a  minimum,  ratings  are 
generated  for  each  of  the  process  area  goals  within  the  appraisal  reference  model  scope.  Ratings 
may  also  be  generated  for  process  areas,  capability  levels,  or  maturity  levels  if  desired  by  the 
appraisal  sponsor.  Maturity  level  ratings  and/or  capability  level  ratings  are  based  on  the 
definitions  of  capability  levels  and  maturity  levels  in  the  appraisal  reference  model.  Refer  to 
Process  Definition  2.6,  Generate  Appraisal  Results  for  additional  information  about  SCAMPI  A 
rating  processes. 

Reporting  Results 

The  results  of  the  appraisal  are  reported  to  the  appraisal  sponsor  and  to  the  appraised  organization. 
In  supplier  selection  or  process  monitoring  contexts,  the  mechanisms  and  timeframe  used  for 
reporting  results  may  be  subject  to  acquisition  or  contractual  restrictions.  An  appraisal  record  is 
generated  and  provided  to  the  sponsor,  documenting  further  information  regarding  the  appraisal 
according  to  the  specifications  found  in  later  sections  of  this  document.  Appraisal  results  are  valid 
for  a  period  not  to  exceed  three  years. 

A  subset  of  this  data  is  provided  to  the  SEI  for  the  purposes  of  quality  control  and  the  collection 
of  appraisal  measures  for  reporting  to  the  appraisal  community.  The  appraisal  data  to  be  provided 
is  defined  by  the  SEI  separately  from  this  document  to  allow  for  continuous  improvement  of 
appraisal  reporting  apart  from  the  CMMI  Product  Suite. 

Instruments  and  Tools 

Instruments  are  artifacts  that  may  be  used  in  an  appraisal  for  the  collection  and  presentation  of 
data.  Instruments  are  provided  by  the  organizational  unit  to  inform  the  appraisal  team  about  the 
processes  implemented  in  the  organization  and  how  they  relate  to  the  appraisal  reference  model. 
Instruments  can  take  various  forms,  including  questionnaires,  surveys,  site  orientation  packets, 
and  mappings  from  reference  model  practices  to  the  corresponding  processes  that  are  defined  and 
implemented  within  the  organizational  unit. 

The  SCAMPI  A  method  does  not  require  that  an  instrument  be  used.  However,  instruments  can 
provide  the  appraisal  team  with  an  in-depth  understanding  of  the  organizational  implementation 
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of  the  model  on  a  practice-level  basis  for  the  basic  units  and  support  functions  within  the 
organizational  unit  to  be  investigated  in  the  appraisal. 

Instruments  also  often  provide  an  opportunity  for  the  organizational  unit  to  provide  a  self¬ 
characterization  of  their  implemented  processes,  identify  applicable  substantiating  objective 
evidence,  and  specify  any  additional  comments  that  might  be  useful  in  understanding  the 
implemented  processes.  Used  in  this  manner,  instruments  can  support  the  SCAMPI  A  method 
emphasis  on  verification-based  appraisals  and  minimize  the  need  for  discovery  of  objective 
evidence  (see  the  discussion  of  Data  Collection  Approaches  on  page  22),  thus  helping  to  facilitate 
efficient  appraisal  performance. 

An  appraisal  team  member  should  facilitate  the  entry  of  data  into  instruments  where  feasible  to 
ensure  that  appropriate  data  are  obtained.  This  approach  can  help  the  appraised  organization 
clarify  or  interpret  the  intent  of  the  appraisal  reference  model  practices,  understand  what  data  are 
expected,  and  focus  the  responses.  The  entry  of  either  too  much  or  too  little  data  into  instruments 
can  be  problematic  for  both  the  appraisal  team  and  the  appraised  organization  and  result  in 
inefficient  use  of  resources. 

Effective  management  of  appraisal  data  is  a  challenge  that  can  be  simplified  with  the  use  of 
automated  tools.  The  SEI  provides  a  rudimentary  toolkit  to  SCAMPI  Lead  Appraisers  that  can  be 
used  to  collect  practice-level  data  and  to  characterize,  consolidate,  and  summarize  infonnation. 
Several  vendor  tools  are  also  available  in  the  marketplace.  The  choice  of  tools  is  largely  one  of 
personal  preference;  some  appraisal  team  leaders  prefer  manual  techniques,  such  as  wall  charts,  to 
record  information  and  findings. 

Conducting  Cost-Effective  Appraisals 

High  appraisal  costs  can  be  a  significant  factor  in  the  overall  business  value  obtained  from 
CMMI-based  process  improvement  and  conducting  SCAMPI  A  appraisals.  While  much  has  been 
done  to  improve  the  integrity  of  maturity  level  ratings  and  protect  the  corresponding  investments 
of  CMMI  adopters,  it  is  the  improvement  actions  taken  as  a  result  of  appraisals,  and  not  the 
benchmark  ratings,  that  are  most  likely  to  provide  substantial  and  lasting  benefits  to  the 
organization’s  business  results  and  operational  effectiveness.  High  appraisal  costs  can 
substantially  impact  the  investment  available  for  other  aspects  of  a  process  improvement  cycle, 
and  could  even  detract  from  a  business  case  for  CMMI  adoption.  We  conclude  this  overview  with 
a  discussion  of  approaches,  both  in  improvements  to  the  SCAMPI  VI. 3  method  and  its  efficient 
enactment  by  adopters,  that  can  help  achieve  cost-effective  SCAMPI  A  appraisals  and  maximize 
business  returns. 

These  descriptions  are  based  on  inefficient  practices  that  have  been  commonly  observed  to  drive 
high  appraisal  costs,  with  a  summary  of  VI. 3  method  improvements  and  recommendations  to  help 
improve  these  practices  and  reduce  appraisal  costs  across  the  community.  Several  of  these 
appraisal  efficiency  topics  are  described  more  fully  in  other  areas  of  the  SCAMPI  A  method 
overview  or  following  sections  of  this  document.  This  is  not  an  exhaustive  list  of  the  sources  of 
appraisal  cost  inefficiencies  or  of  the  potential  strategies  that  could  help  address  them;  numerous 
ideas  are  also  suggested  in  other  publications  and  conference  presentations. 
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Strategies  for  Planning  Cost-Effective  SCAMPI  Appraisals 

Data  Collection 

(For  more  detailed  information,  also  refer  to  the  Data  Collection  topic  on  page  19.) 

Observation:  Inefficient  collection  of  data  from  the  appraised  organization  has  been  a  driver  of 
high  appraisal  costs.  Much  of  this  has  been  due  to  a  misunderstanding  of  Verification-mode 
appraisals,  which  are  an  optional  approach  (not  a  SCAMPI  A  method  requirement)  intended  to 
save  costs  by  leveraging  any  existing  assets  the  organization  may  be  able  to  provide  to  help  the 
data  collection  effort.  Instead,  this  has  often  led  to  a  disproportional  shift  of  data  collection 
responsibility  to  the  appraised  organization.  Efforts  to  build  model-centric  evidence  tables  (e.g. 
PIIDs,  populated  with  both  direct  and  indirect  artifacts)  can  increase  appraisal  preparation  costs. 

Method:  Revisions  made  to  the  method  clarified  and  contrasted  data  collection  approaches.  It 
also  added  a  focus  on  managed  discovery,  phased  data  collection,  and  product-centric  data  sets  to 
reduce  costs  associated  with  data  collection  and  better  balance  effort  between  the  organization  and 
the  appraisal  team.  The  revisions  to  the  method  also  added  emphasis  on  maintaining  a  data 
collection  plan  to  anticipate,  reduce,  and  manage  the  data  collection  effort,  in  communication  with 
the  appraisal  sponsor. 

Recommendations:  Negotiate  expectations  for  the  delivery  of  data  to  the  appraisal  team.  Lead 
appraisers  are  trained  to  seek  artifacts  that  result  from  routine  implementation  of  practices.  Use 
managed  discovery  and  other  techniques  to  enable  the  appraisal  team  to  incrementally  request  and 
map  specific  artifacts  to  the  model.  Iterative  data  collection  allows  the  team  to  ask  for  the  only  the 
data  that  are  needed  thereby  optimizing  the  use  of  resources  by  the  appraised  organization.  Work 
with  the  lead  appraiser  to  strike  a  balance  between  the  duration  of  the  appraisal  and  the  time  spent 
preparing  for  it  in  order  to  balance  data  collection  time  between  the  two. 

Sampling  Factors 

Observation:  Prior  SCAMPI  versions  were  based  on  focus  and  non-focus  projects,  with  at  least 
three  instantiations  needed  for  each  process  area,  and  representation  of  the  organizational  unit 
based  on  coverage  of  critical  factors.  Without  guidance  about  the  number  of  projects  and 
instances  needed,  implementation  on  appraisals  has  sometimes  been  inconsistent,  impacting 
repeatability,  confidence  in  the  representativeness  of  the  sample,  and  uncertainty  about  the 
amount  of  data  collection  needed  (too  much  or  too  little)  which  can  impact  predictability  of 
appraisal  costs. 

Method:  Quantitative  sampling  approaches  have  been  implemented  (described  in  detail  elsewhere 
in  the  MDD)  to  determine  the  organizational  scope  based  on  sampling  factors  and  subgroups 
reflecting  coverage  of  the  variety  of  conditions  under  which  the  process  is  used.  These  sampling 
factors,  in  conjunction  with  data  coverage  mles,  define  how  much  data  is  needed  to  assure 
coverage.  Not  only  does  this  improve  representation  and  confidence  in  results,  but  selection  of 
well-designed  sampling  factors  may  reduce  the  overall  amount  of  data  needed  and  associated  cost 
incurred  by  the  appraised  organization. 
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Recommendations:  Invest  due  diligence  in  defining  sampling  factors  that  truly  characterize  the 
range  of  conditions  in  which  the  process  must  operate.  Work  with  the  lead  appraiser  to  find  a 
value-added  compromise  between  depth  and  breadth.  The  lead  appraiser  will  help  the  sponsor  to 
strike  a  balance  between  generalizability  and  appraisal  cost.  If  the  initial  appraisal  scope  is  too 
costly,  consider  constraining  the  organizational  scope  to  a  smaller  subset  of  the  organization 
where  the  attributed  appraisal  results  (e.g.,  maturity  level  ratings)  are  most  valuable  and  needed. 

Understand  SCAMPI  Tailoring  Options 

Observation:  For  both  lead  appraisers  and  sponsoring  organizations,  a  lack  of  clarity  about  the 
range  of  accepted  tailoring  options  may  have  led  to  an  over-reliance  on  traditionally  implemented 
approaches. 

Method:  This  revised  method  clarifies  tailoring  options  and  the  range  of  expected 
implementations  of  the  method. 

Recommendations:  Lead  appraisers  serve  as  an  intermediary  for  appraisal  sponsors  and 
organizations,  and  when  needed,  act  as  a  translator  between  business  space  and  model  space. 
Appraisal  sponsors,  as  informed  consumers  should  strive  to  understand  available  SCAMPI 
tailoring  options  and  make  trade-offs  to  help  conduct  cost-effective  appraisals. 

Method  Description 

This  section  provides  an  overview  of  the  SCAMPI  A  method  architecture,  including  appraisal 
phases,  processes,  and  activities.  These  descriptions  are  high-level  abstractions  of  the  process 
descriptions  contained  in  Part  II  of  this  document. 

A  summary  of  the  SCAMPI  A  method  processes  and  activities  for  each  of  the  three  appraisal 
phases  is  contained  in  Table  9,  Table  10,  and  Table  11. 
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Table  9:  SCAMPI  A  Phase  Summary:  Plan  and  Prepare  for  Appraisal 


Phase 

Process 

Purpose 

Activities 

1  Plan  and 

1.1 

Analyze 

Understand  the  business  needs 

1.1.1 

Determine  Appraisal 

Prepare  for 

Requirements 

of  the  organizational  unit  for 

Objectives 

Appraisal 

which  the  appraisal  is  being 

1.1.2 

Determine  Data 

requested.  The  appraisal  team 

Collection  Strategy 

leader  will  collect  information  and 

1.1.3 

Determine  Appraisal 

help  the  appraisal  sponsor  match 

Constraints 

appraisal  objectives  with  their 

1.1.4 

Determine  Appraisal 

business  objectives. 

1.1.5 

Scope 

Determine  Appraisal 
Outputs 

1.1.6 

Obtain  Commitment  to 
Initial  Appraisal  Plan 

1.2 

Develop 

Document  the  results  of 

1.2.1 

Tailor  Method 

Appraisal  Plan 

appraisal  planning  including  the 

1.2.2 

Identify  Needed 

requirements,  agreements, 

Resources 

estimates,  risks,  method 

1.2.3 

Develop  Data 

tailoring,  and  practical 

Collection  Plan 

considerations  (e.g.,  schedules, 

1.2.4 

Determine  Cost  and 

logistics,  and  contextual 

Schedule 

information  about  the 
organization)  associated  with  the 

1.2.5 

Plan  and  Manage 
Logistics 

appraisal.  Obtain  and  record  the 
sponsor’s  approval  of  the 
appraisal  plan. 

1.2.6 

1.2.7 

Document  and 

Manage  Risks 

Obtain  Commitment  to 
Appraisal  Plan 

1.3 

Select  and 

Ensure  that  an  experienced, 

1.3.1 

Identify  Appraisal 

Prepare  Team 

objective,  trained,  and 

Team  Leader 

appropriately  qualified  team  is 

1.3.2 

Select  Team  Members 

available  and  prepared  to 

1.3.3 

Document  and 

execute  the  appraisal  process. 

1.3.4 

Manage  Conflicts  of 
Interest 

Prepare  Team 

1.4 

Obtain  and 

Obtain  information  that  facilitates 

1.4.1 

Obtain  Initial  Objective 

Inventory  Initial 

site-specific  preparation  and  an 

Evidence 

Objective 

understanding  of  the 

1.4.2 

Inventory  Objective 

Evidence 

implementation  of  model 
practices  across  the 
organizational  unit.  Identify 
potential  issues,  gaps,  or  risks  to 
aid  in  refining  the  plan. 

Strengthen  the  appraisal  team 
members’  understanding  of  the 
organization’s  operations  and 
processes. 

Evidence 
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Phase 

Process 

Purpose 

Activities 

1.5  Prepare  for 
Appraisal 
Conduct 

Ensure  readiness  to  conduct  the 
appraisal,  including  confirmation 
of  the  availability  of  objective 
evidence,  appraisal  team 
commitment,  logistics 
arrangements,  risk  status  and 
associated  mitigation  plans.  Plan 
and  document  data  collection 
strategies. 

1.5.1  Perform  Readiness 

Review 

1.5.2  Re-Plan  Data 

Collection 

Table  10:  SCAMPI  A  Phase  Summary:  Conduct  Appraisal 


Phase 

Process 

Purpose 

Activities 

2  Conduct 
Appraisal 

2.1  Prepare 
Participants 

Ensure  that  appraisal 
participants  are  appropriately 
informed  of  the  appraisal 
process,  purpose,  and 
objectives  and  are  available  to 
participate  in  the  appraisal 
process. 

2.1.1  Conduct  Participant 

Briefing 

2.2  Examine 
Objective 
Evidence 

Examine  information  about  the 
practices  implemented  in  the 
organization  and  relate  the 
resultant  data  to  the  appraisal 
reference  model.  Perform  the 
activity  in  accordance  with  the 
data  collection  plan.  Take 
corrective  actions  and  revise 
the  data  collection  plan  as 
needed. 

2.2.1  Examine  Objective 

Evidence  from  Artifacts 

2.2.2  Examine  Objective 

Evidence  from  Affirmations 

2.3  Document 
Objective 
Evidence 

Create  lasting  records  of  the 
information  gathered  by 
identifying  and  then 
consolidating  notes, 
transforming  the  data  into 
records  that  document  gaps  in 
practice  implementation  or 
exemplary  practice 
implementation. 

2.3.1  Take/Review/Tag  Notes 

2.3.2  Record  Presence/Absence 
of  Objective  Evidence 

2.3.3  Document  Practice 
Implementation 

2.3.4  Review  and  Update  the 

Data  Collection  Plan 
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Phase 

Process 

Purpose 

Activities 

2.4  Verify 
Objective 
Evidence 

Verify  the  sufficiency  of 
objective  evidence  to 
determine  the  implementation 
of  model  practices  for  each 
instantiation.  Describe  any 
strengths  and  weaknesses  in 
the  implementation  of  model 
practices.  Each  implementation 
of  each  practice  is  verified  so 
that  it  may  be  compared  to  the 
practices  of  the  reference 
model.  Then  the  team 

characterizes  the  extent  to 
which  the  practices  in  the 
model  are  implemented. 

2.4.1  Verify  Objective  Evidence 

2.4.2  Characterize 

Implementation  of  Model 
Practices  and  Generate 
Preliminary  Findings 

2.5  Validate 
Preliminary 
Findings 

Validate  preliminary  findings, 
including  weaknesses  (i.e., 
gaps  in  practice 
implementation)  and  strengths 
(i.e.,  exemplary  implementation 
of  model  practices)  with 
members  of  the  organizational 
unit. 

2.5.1  Validate  Preliminary 

Findings 

2.6  Generate 
Appraisal 
Results 

Rate  goal  satisfaction  based  on 
the  extent  of  practice 
implementation  throughout  the 
organizational  scope  of  the 
appraisal.  The  extent  of 
practice  implementation  is 
judged  based  on  validated  data 
(e.g.,  artifact  and  affirmation 
objective  evidence)  collected 
from  the  entire  representative 
sample  of  the  organizational 
unit.  Aggregate  ratings 
(process  area  ratings,  maturity 
level  ratings,  capability  ratings, 
etc.)  are  driven  by  the  goal 
satisfaction  ratings. 

2.6.1  Derive  Findings  and  Rate 
Goals 

2.6.2  Determine  Process  Area 
Ratings 

2.6.3  Determine  Process  Area 

Profile 

2.6.4  Determine  Maturity  Level 

2.6.5  Document  Appraisal 

Results 
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Table  1 1:  SCAMPI  A  Phase  Summary:  Report  Results 


Phase 

Process 

Purpose 

Activities 

3  Report 

3.1  Deliver 

Provide  credible  appraisal  results 

3.1.1  Deliver  Final  Findings 

Results 

Appraisal 

Results 

that  can  be  used  to  guide  actions. 
Represent  the  strengths  and 
weaknesses  of  the  processes  in  use 
at  the  time.  Provide  ratings  (if 
planned  for)  that  accurately  reflect 
the  capability  level  or  maturity  level 
of  the  processes  in  use. 

3.1.2  Conduct  Executive 
Session(s) 

3.1.3  Plan  for  Next  Steps 

3.2  Package  and 
Archive 
Appraisal 
Assets 

Preserve  important  data  and 
records  from  the  appraisal,  and 
dispose  of  sensitive  materials  in  an 
appropriate  manner. 

3.2.1  Collect  Lessons 

Learned 

3.2.2  Generate  Appraisal 
Record 

3.2.3  Provide  Appraisal 
Feedback  to  the  SEI 

3.2.4  Archive  and/or  Dispose 
of  Key  Artifacts 
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Part  II:  Process  Definitions 
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1  Plan  and  Prepare  for  Appraisal 


1.1  Analyze  Requirements 


Purpose 

Understand  the  business  needs  of  the  organization  for  which  the  appraisal  is 
being  requested.  The  appraisal  team  leader  will  collect  information  and  help  the 
appraisal  sponsor  match  appraisal  objectives  with  their  business  objectives. 

Detennine  and  communicate  the  strategy  for  collecting  appraisal  evidence.  The 
appraisal  team  leader  will  work  with  the  appraisal  sponsor  to  determine  the 
overall  strategy  for  collecting  appraisal  infonnation.  This  strategy  will  form  the 
basis  for  the  appraisal  data  collection  plan. 

Entry  Criteria 

•  An  appraisal  sponsor  has  decided  that  a  SCAMPI  appraisal  should  be 
performed. 

•  People  who  can  provide  statements  of  requirements  for  the  appraisal  are 
available. 

Inputs 

•  sponsor  appraisal  objectives 

•  initial  requirements  and  constraints 

•  process-related  legacy  information 

•  business  objectives 

Activities 

1.1.1  Determine  Appraisal  Obj  ectives 

1.1.2  Determine  Data  Collection  Strategy 

1.1.3  Determine  Appraisal  Constraints 

1.1.4  Determine  Appraisal  Scope 

1.1.5  Determine  Appraisal  Outputs 

1.1.6  Obtain  Commitment  to  Initial  Appraisal  Plan 

Outputs 

Initial  Appraisal  Plan 

Outcome 

The  decision  to  proceed  with  the  appraisal  based  on  a  shared  understanding  of 
the  appraisal  objectives,  constraints,  outputs,  and  scope. 
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1.1  Analyze  Requirements 


Exit  Criteria 

•  Initial  contact  between  the  appraisal  sponsor  and  the  appraisal  team  leader 
(i.e.,  a  certified  or  candidate  SCAMPI  Lead  Appraiser)  has  occurred. 

•  The  appraisal  team  leader  has  been  given  access  to  members  of  the 
sponsoring  organization. 

•  The  initial  strategy  for  collecting  appraisal  data  has  been  established  for  use 
in  subsequent  planning  activities. 

•  The  initial  appraisal  plan  has  been  approved  by  the  appraisal  sponsor  and 
placed  under  change  management. 

Key  Points 

At  this  early  stage  in  the  process,  gathering  information  that  supports  good 
planning  is  most  important.  Often,  the  appraisal  team  leader  must  educate 
members  of  the  sponsor’s  organization  in  the  purpose  and  role  of  appraisals. 

Tools  and 
Techniques 

Collaborative  consultation  between  the  appraisal  team  leader  and  the  appraisal 
sponsor  is  important  in  this  activity.  The  appraisal  team  leader  may  be  able  to 
simply  interview  the  sponsor  to  get  the  needed  information  and  reach 
agreements.  In  some  settings,  a  series  of  meetings  with  different  stakeholders 
may  be  needed  to  elicit  and  build  consensus  on  the  business  needs  that  can  be 
met  through  a  SCAMPI  A  appraisal. 

Understanding  the  history  of  appraisals  in  the  organization,  especially  the 
organizational  and  appraisal  reference  model  scope  of  past  appraisals,  is 
important  for  understanding  the  requirements  for  the  appraisal  under 
consideration.  The  choices  sponsors  make  about  appraisal  scope  are  often  tied  to 
their  (sometimes  unstated)  priorities  for  process  improvement. 

Metrics 

A  number  of  metrics  support  the  appraisal  team  leader’s  monitoring  of  this  work: 

•  calendar  time  between  initial  contact  and  finalization  of  requirements 

•  effort  expended  to  gather  and  analyze  requirements 

•  number  of  meetings  with  representatives  of  the  sponsoring  and/or  appraised 
organization 

Verification  and  The  exit  criterion  for  this  activity  is  the  approval  of  the  initial  appraisal  plan  and 


Validation 

its  placement  under  change  management. 

Review  of  the  documented  agreements  resulting  from  the  work  of  this  set  of 
activities  will  serve  to  validate  the  requirements,  which  feed  into  appraisal 
planning. 

Records 

The  initial  appraisal  plan 
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1.1  Analyze  Requirements 


Interfaces 
with  Other 
Processes 


This  process  is  a  foundation  for  the  success  or  failure  of  the  entire  appraisal;  it 
is  at  this  point  in  the  appraisal  that  the  most  leverage  exists  for  avoiding 
problems  and  issues  downstream.  Gathering  and  understanding  the 
requirements  for  the  conduct  of  a  SCAMPI  A  appraisal  is  vital  to  making 
appropriate  decisions  and  providing  value  to  the  sponsor.  Many  examples  of 
problems  encountered  during  appraisals  can  be  traced  to  shortcomings  in  the 
conduct  of  this  process.  The  activities  described  here  form  the  basis  for  the 
activities  described  in  the  next  process.  Develop  Appraisal  Plan. 

The  selected  data  collection  strategy,  which  includes  the  data  collection 
approach  (discovery,  managed  discovery  and/or  verification)  will  affect  the 
activities  of  Section  1.4,  Obtain  and  Inventory  Initial  Objective  Evidence.  The 
balance  between  discovery  and  verification  will  have  a  significant  impact  on  the 
amount  of  time  required  by  the  organization  to  prepare  for  an  appraisal  in  a 
verification-based  mode  or  the  amount  of  time  for  the  team  to  conduct  the 
appraisal  in  a  discovery-based  mode.  A  managed  discovery  approach  balances 
verification  and  discovery  activities  using  an  iterative,  phased  based  approach 
to  data  collection.  The  results  of  the  readiness  review  in  Section  1.5,  Prepare  for 
Appraisal  Conduct  could  result  in  revisions  to  the  data  collection  approach. 


Summary  of 
Activities 


The  objectives  that  motivate  the  conduct  of  an  appraisal  as  well  as  the  intended 
strategy  for  collecting  appraisal  data  must  be  well  understood  so  that 
appropriate  participants,  tailoring  decisions,  and  appraisal  outputs  can  be 
selected.  The  constraints  that  shape  the  appraisal  enactment,  in  light  of  the 
objectives  and  data  collection  strategies,  may  limit  achievement  of  the  desired 
result  if  they  are  not  adequately  understood  and  negotiated.  A  clear  agreement 
regarding  appraisal  outputs  and  their  intended  usage  will  help  maintain  the 
sponsorship  needed  for  conducting  the  appraisal  and  acting  on  the  results. 
Establishing  agreement  on  these  objectives,  constraints,  outputs,  and  intended 
usage  forms  the  basis  for  a  commitment  to  the  plan  for  conducting  the 
appraisal. 
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1.1.1  Determine  Appraisal  Objectives 


Activity 

Description 


Required 

Practices 


Parameters  and 
Limits 


The  business  needs  for  process  improvement  drive  the  requirements  for  the 
conduct  of  any  given  appraisal  and  generally  include  one  or  more  of  three 
closely  related  factors: 

•  reducing  costs 

•  improving  quality 

•  decreasing  time  to  market 

The  fundamental  premise  of  process  improvement  is  that  organizational 
processes  significantly  impact  these  factors. 

Obtaining  a  fair  and  objective  characterization  of  the  processes  in  use  in  the 
organization(s)  is  the  essential  reason  for  conducting  an  appraisal.  In  addition  to 
this  motivation,  a  sponsor’s  desire  to  conduct  an  appraisal  could  be  driven  by 
one  or  more  of  the  following  business-related  objectives: 

•  Document  a  credible  benchmark  that  reflects  successful  process 
improvement. 

•  Evaluate  areas  of  potential  risk  that  may  affect  the  performance  of  the 
organization. 

•  Involve  members  of  the  appraised  organization  in  improving  the 
performance  of  the  process. 

•  Support  specific  decisions  related  to  the  direction  of  a  new  or  existing 
improvement  program. 

•  Motivate  a  supplier  to  focus  on  process  issues  that  affect  contract 
performance. 


The  appraisal  team  leader  or  designee  shall 

•  identify  sponsor  and  relevant  stakeholders,  and  establish  communication 

•  document  the  business  objectives  provided  by  the  sponsor  and  the  specific 
appraisal  objectives 

•  ensure  the  alignment  of  the  appraisal  objectives  to  the  business  objectives 

•  determine  and  document  the  appraisal  usage  mode  (i.e.,  Internal  Process 
Improvement,  Supplier  Selection,  or  Process  Monitoring) 


At  least  one  communication  between  the  appraisal  team  leader  and  sponsor  is 
required. 
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1.1.1  Determine  Appraisal  Objectives 


Implementation 

Guidance 


Organizations  with  experience  in  the  use  of  appraisals  may  have  a  clear  set  of 
appraisal  objectives  identified  in  advance  of  contacting  an  appraisal  team 
leader. 

In  some  cases,  the  usage  mode  will  be  self-evident;  however,  there  may  be 
instances  in  which  the  appraisal  sponsor  either  may  not  be  sure  or  may  have 
made  an  assumption  that  is  not  founded  on  fact.  The  appraisal  team  leader  is 
responsible  for  ensuring  that  the  best  choice  of  usage  mode  is  made  consistent 
with  the  sponsor’s  input  and  direction. 

Depending  on  the  structure  of  the  appraised  organization,  as  well  as  the  usage 
mode,  it  is  often  important  to  distinguish  the  role  of  senior  site  manager  from 
that  of  the  appraisal  sponsor.  For  some  appraisals,  these  two  roles  are 
encompassed  in  the  duties  of  a  single  person.  For  other  appraisals,  these  two 
roles  may  represent  two  people  working  many  time  zones  away  from  each 
other. 


♦ 


CMU/SEI-2011-HB-001  |  41 


1.1.2  Determine  Data  Collection  Strategy 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  data  collection  strategy  is  determined  based  on  the  appraisal  objectives. 

The  data  collection  strategy  employed  to  obtain  an  objective  characterization  of 
the  processes  in  use  in  the  organization  has  major  implications  for  the  appraisal 
in  terms  of: 

•  the  amount  of  time  and  effort  expended  by  the  organization  in  preparing  for 
the  appraisal 

•  the  ability  of  the  team  to  make  accurate  judgments 

•  the  usefulness  and  accuracy  of  the  appraisal  results 

•  the  overall  cost  of  the  conduct  appraisal  phase 

A  well-defined  data  collection  strategy  is  important  for  appraisal  planning,  as  it 
provides  the  basis  for  detailed  data  collection  planning  (see  activity  1.5,  Prepare 
for  Appraisal  Conduct)  and  examining  objective  evidence  (see  activity  2.2, 
Examine  Objective  Evidence).  It  should  be  established  early  and  continually 
refined  throughout  the  Plan  and  Prepare  phase  as  it  provides  the  basis  for 
effective  data  collection. 

The  data  collection  strategy  outlines  the  overall  high  level  scheme  for  data 
collection  including  the  choice  of  data  collection  approach  (discovery,  managed 
discovery,  and/or  verification),  when  the  data  will  be  collected  (e.g., 
preparation  phase  or  conduct  phase)  and  what  data  collection  techniques  (e.g., 
demonstrations,  presentations,  interviews  and  questionnaires)  will  be  employed 
for  both  objective  evidence  types  (artifacts  and  affirmations).  A  complete  data 
collection  strategy  will  address  how  and  when  all  evidence  types  will  be 
collected  (artifacts  and  affirmations).  It  will  specify  what  the  organization  is 
responsible  for  collecting. 


The  appraisal  team  leader  or  designee  shall 

•  work  with  the  sponsor  or  designee  to  identify  the  data  collection  strategy 
that  best  aligns  with  appraisal  objectives  and  constraints 

•  document  the  planned  data  collection  strategy 

•  document  the  tailoring  decisions  made  using  the  tailoring  checklist  in 
Appendix  H  -  SCAMPI  A  Tailoring  Checklist 


The  data  collection  strategy  must  address: 

•  the  data  collection  approach  (discovery,  managed  discovery  and/or 
verification) 

•  the  techniques  for  collecting  artifacts  (e.g.,  documents,  demonstrations, 
presentations) 

•  the  techniques  for  collecting  affirmations  (e.g.,  interviews,  demonstrations, 
presentations) 

•  a  high-level  schedule  for  collecting  data 

•  roles  and  responsibilities  for  collecting  appraisal  data 
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1.1.2  Determine  Data  Collection  Strategy 


Parameters 
and  Limits 


Implementation 

Guidance 


The  identified  data  collection  strategy  must  be  documented  in  the  data 
collection  plan,  which  is  part  of  the  overall  appraisal  plan.  The  data  collection 
strategy  will  evolve  throughout  the  appraisal  planning  process. 


There  are  three  basic  data  collection  approaches. 

Discovery  -  A  data  collection  approach  where  limited  objective  evidence  is 
provided  by  the  appraised  organization  prior  to  the  appraisal,  and  the  appraisal 
team  must  probe  and  uncover  a  majority  of  the  objective  evidence  necessary  to 
obtain  sufficient  coverage  of  reference  model  practices. 

Managed  Discovery  —  A  phased  data  collection  approach  beginning  with  an 
initial  data  call  for  a  pre-determined  set  of  artifacts  (e.g.,  plans,  requirements 
documentation,  test  procedures),  followed  by  a  set  of  iterative  calls  based  on 
the  appraisal  team’s  evaluation  of  the  artifacts  and  remaining  evidence  gaps. 
Refer  to  Appendix  E,  Managed  Discovery  for  more  information  on  managed 
discovery  and  comparison  to  discovery  and  verification  data  collection 
approaches. 

Verification  -  A  data  collection  approach  in  which  the  focus  of  the  appraisal 
team  is  on  verifying  the  set  of  objective  evidence  provided  by  the  appraised 
organization  (i.e.  mapped  to  CMMI  practices)  in  advance  of  the  appraisal. 

Appraisal  team  leaders,  in  conjunction  with  the  appraisal  sponsor,  select  one 
or  a  combination  of  these  three  data  collection  approaches  to  best  satisfy  the 
needs  of  the  organization  and  appraisal. 

Organizations  that  have  not  previously  conducted  an  appraisal  or  are 
increasing  their  organizational  maturity  or  capability  relative  to  a  previous 
appraisal  may  likely  find  a  discovery  or  managed  discovery  data  collection 
approach  (at  least  in  part)  would  make  the  most  effective  use  of  organizational 
resources  to  prepare  for  the  appraisal.  These  two  discovery-type  approaches 
would  minimize  organizational  resources  to  supply  artifacts  that  are  not 
applicable  and/or  are  never  reviewed  by  the  appraisal  team  due  to  the 
organization  misinterpreting  the  CMMI  model  and/or  the  misperceiving 
appraisal  team’s  need  for  artifacts.  Organizations  that  employ  a  discovery  or 
managed  discovery  data  collection  approach  are  likely  to  expend  far  less 
resources  in  preparing  data  for  the  appraisal  than  if  a  verification  data 
collection  approach  is  used. 

Organizations  that  have  undergone  past  appraisals  may  have  a  data  collection 
approach  the  appraisal  team  can  use.  If  so,  a  verification-based  appraisal  may 
be  the  most  efficient  data  collection  approach.  The  appraisal  team  lead  should 
work  with  organization  staff  to  ensure  that  existing  strategies  are  still 
applicable,  and  changes  should  be  made  to  ensure  that  the  strategy  aligns  well 
with  the  objectives  and  usage  mode  of  the  current  appraisal. 
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1.1.2  Determine  Data  Collection  Strategy 


Implementation 

Guidance 

(continued) 


Organizations  that  have  undergone  past  appraisals  typically  have  an  existing 
objective  evidence  data  base  mapped  to  model  practices.  The  use  of  prior 
appraisal  events  (e.g.,  SCAMPI  B  or  SCAMPI  C  appraisals)  can  be  used  to 
help  complete  the  organization’s  evidence  database,  and  can  be  included  in  the 
defined  data  collection  strategy.  The  data  collection  strategy  will  impact 
appraisal  planning,  and  may  be  revised  after  planning  milestones,  such  as 
readiness  reviews,  other  appraisal  events,  sponsor  meetings,  or  even  changes  in 
business  objectives.  As  changes  or  refinements  are  determined,  the  data 
collection  plan  will  be  maintained  to  accurately  reflect  the  strategy  in  place. 


The  data  collection  strategy  should  be  tailored  to  meet  the  needs  of  the 
organization  and  the  appraisal  team.  It  is  important  that  the  strategy  be 
documented  and  understood,  so  that  appropriate  appraisal  planning  can  be 
done. 


♦ 
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1.1.3  Determine  Appraisal  Constraints 


Activity 

Description 

The  constraints  within  which  the  appraisal  must  be  conducted  are  determined 
based  on  a  dialog  between  the  appraisal  team  leader  and  the  appraisal  sponsor 
and/or  senior  site  manager.  This  dialog  typically  is  an  iterative  process  in  which 
the  preferences  of  the  appraisal  sponsor,  the  limits  of  the  method,  and  the 
consequent  resource  requirements  are  balanced  against  each  other  to  arrive  at 
an  optimal  set  of  appraisal  plan  parameters. 

Required 

Practices 

The  appraisal  team  leader  or  designee  shall 

•  establish  high-level  cost  and  schedule  constraints 

•  document  model  scope  of  the  appraisal  and  initial  rating  targets  (if  any) 

•  document  initial  description  of  the  organizational  unit 

•  negotiate  constraints  and  objectives  with  stakeholders  to  ensure  feasibility 

•  document  negotiated  constraints  to  be  met 

Appraisal  constraints  are  documented  in  the  appraisal  plan. 

Parameters 
and  Limits 

Constraints  identified  by  the  appraisal  plan  must  be  negotiated  between  the 
sponsor  and  the  appraisal  team  leader. 

Implementation 

Guidance 

Constraints  on  cost  and  schedule  identified  during  this  early  stage  of  the 
appraisal  are  expected  to  be  high  level  and  not  detailed  estimates.  They  may 
take  the  form  of  statements  such  as  “We  need  this  to  be  done  in  Q4,”  “You 
can’t  use  more  than  five  of  my  people  on  the  team,”  and  “I  can’t  afford  to  have 
it  last  more  than  a  month.”  During  these  initial  discussions  with  the  sponsor, 
the  appraisal  team  leader  gains  an  understanding  of  the  desired  model  scope  as 
well  as  the  organizational  scope  of  the  appraisal.  The  process  of  specifying  the 
organizational  unit,  as  well  as  the  sampling  that  defines  the  organizational 
scope  of  the  appraisal  (described  in  activity  1.1.4,  Determine  Appraisal  Scope) 
is  influenced  by  this  understanding. 

Practical  limitations  relating  to  time,  cost,  and  effort  are  clarified  and 
negotiated  in  the  context  of  other  requirements  the  sponsor  has.  The  business 
context  in  which  the  appraisal  is  conducted  drives  choices  that  the  appraisal 
team  leader  must  make.  For  example,  if  virtual  methods  (e.g.,  video 
conferences,  teleconferences,  and  other  similar  technology)  are  to  be  used  to 
conduct  appraisal  activities,  the  constraints  imposed  by  these  methods  should 
be  discussed,  documented,  and  taken  into  account  as  the  appraisal  is  planned. 
Appraisals  should  not  be  conducted  in  isolation  from  other  activities  relating  to 
process  management  and  improvement.  The  needs  of  relevant  stakeholders,  be 
they  acquisition  organizations  or  division  heads  managing  an  engineering- 
related  process  group,  often  place  requirements  on  the  conduct  of  the  appraisal. 

♦ 
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1.1.4  Determine  Appraisal  Scope 


Activity  The  appraisal  scope  consists  of  the  appraisal  reference  model  scope  and  the 

Description  organizational  scope  to  be  examined  during  the  appraisal.  The  reference  model  scope 
must  be  determined  and  documented  early  in  the  planning  process,  identifying  the 
relevant  models  and  process  areas. 

For  CMMI  models,  the  scope  specification  includes  selection  of  the  staged 
representation  (and  the  maturity  levels  included)  or  the  continuous  representation 
(and  the  capability  levels  included).  In  conjunction  with  the  appraisal  sponsor,  the 
appraisal  team  leader  is  responsible  for  deciding  which  process  areas  to  include  in  the 
scope  of  the  appraisal  and  which  model  representation  to  use. 

An  organizational  unit  is  the  part  of  an  organization  that  is  the  subject  of  an  appraisal 
and  to  which  the  appraisal  results  will  be  generalized.  This  organizational  unit  may 
include  the  entire  organization,  one  or  more  divisions  within  the  organization,  or  one 
or  more  basic  units  and  support  functions  within  the  organization.  Depending  on  the 
reference  model  used  for  the  appraisal,  the  organizational  unit  may  be  comprised  of 
different  types  of  basic  units  and  support  functions.  These  building  blocks  of  the 
organizational  unit  will  differ  according  to  how  people  are  organized  to  accomplish 
the  work.  The  organizational  scope  is  a  subset  of  the  organizational  unit.  The 
organizational  scope  is  determined  by  selecting  support  functions  and  sampling  basic 
units  to  supply  data  for  the  appraisal. 

The  organizational  scope  of  the  appraisal  is  selected  as  a  representative  sample  of  the 
organizational  unit,  based  on  sampling  factors  that  reflect  meaningful  differences  in 
the  conditions  under  which  work  is  performed.  This  selection  process  involves 
defining  subgroups  that  reflect  differences  according  to  the  sampling  factors.  Basic 
units  within  these  subgroups  are  sampled  in  the  process  of  defining  the  organizational 
scope. 

There  can  be  an  iterative  process  in  which  a  preliminary  specification  of  the 
organizational  scope  leads  to  re-scoping  the  organizational  unit.  A  more  narrowly 
defined  organizational  unit  permits  a  smaller  organizational  scope  of  the  appraisal. 
Conversely,  a  broader  definition  of  the  organizational  unit  leads  to  a  larger 
organizational  scope  of  the  appraisal.  This  interplay  between  scoping  and  sampling  is 
managed  collaboratively  by  the  appraisal  team  leader  and  the  appraisal  sponsor. 

Finally,  the  mapping  of  process  areas  to  basic  units  and  support  functions  is 
established  to  support  data  collection  planning.  Differences  in  the  way  people  are 
organized  to  perform  the  work  within  the  organizational  unit  may  affect  the  sources 
of  data  which  must  be  considered  to  appraise  each  process  area.  Some  organizations 
may  share  resources  to  perform  support  functions  like  configuration  management  or 
measurement  and  analysis  whereas  other  organizations  may  establish  those  functions 
within  each  basic  unit.  Data  coverage  requirements  applied  to  the  mapping  between 
the  model  and  the  organization  form  the  detailed  data  collection  plan. 
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1.1.4  Determine  Appraisal  Scope 


Required 

Practices 


Parameters 
and  Limits 


The  appraisal  team  leader,  in  conjunction  with  the  appraisal  sponsor  and/or  the 

sponsor’s  designee,  shall  determine  and  document  the  following: 

•  the  reference  model  scope  to  be  used  for  the  appraisal 

•  the  organizational  unit  to  be  investigated  during  the  appraisal 

•  the  basic  units  into  which  people  are  organized  to  accomplish  the  work  done 
by  the  organizational  unit 

•  the  support  functions  that  exist  and  the  responsibilities  assigned  to  each 
function 

•  the  sampling  factors  used  to  specify  the  organizational  unit  and  the 
organizational  scope  of  the  appraisal 

•  a  mapping  between  each  basic  unit  or  support  function  and  the  process  areas 
in  the  scope  of  the  appraisal 

•  the  organizational  scope  of  the  appraisal 

•  the  list  of  individuals  who  will  participate  in  the  appraisal 

•  the  planned  coverage  of  each  process  area  in  the  model  scope  of  the 
appraisal  for  each  basic  unit  or  support  function  within  the  organizational 
scope  of  the  appraisal 


The  reference  model  scope  shall  include  the  process  areas  in  the  selected 
model(s)  and  for  CMMI  models  the  representation  chosen  as  well  as  the 
associated  maximum  capability  or  maturity  level  targeted. 

The  reference  model  scope  of  the  appraisal  shall  include  at  least  one  process 
area.  All  practices  and  goals  that  are  part  of  the  selected  process  areas  must  be 
included;  individual  practices  and  goals  within  a  process  area  shall  not  be 
excluded. 

When  a  process  area  is  detennined  to  be  outside  of  the  organizational  unit's 
scope  of  work,  the  process  area  is  designated  as  “not  applicable.”  Any  process 
area  designated  as  “not  applicable”  and  the  rationale  for  its  exclusion  must  be 
documented  in  the  appraisal  plan  and  appraisal  disclosure  statement  (ADS). 
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1.1.4  Determine  Appraisal  Scope 


Parameters 
and  Limits 
(continued) 


The  organizational  scope  of  the  appraisal  shall  be  documented  by  listing  the 
basic  units  and  support  functions  selected  for  inclusion  in  the  appraisal.  The 
rationale  for  these  selections  is  based  on  the  way  the  organization  allocates  roles 
and  responsibilities  to  accomplish  the  work  as  well  as  the  sampling  factors  that 
affect  the  way  people  do  the  work. 

Sampling  factors  associated  with  the  variety  and  diversity  of  conditions  under 
which  work  is  performed  in  the  organizational  unit  must  be  recorded  during 
planning.  The  following  candidate  sampling  factors  must  be  evaluated  to 
determine  the  organizational  scope  of  the  appraisal: 

•  Location:  if  work  is  perfonned  differently  in  different  locations  (e.g., 
countries,  cities,  sites  or  installations) 

•  Customer:  if  work  is  performed  differently  depending  on  the  customer 
served  by  that  work 

•  Size:  if  work  is  performed  differently  based  on  the  size  of  the  basic  unit  or 
support  function 

•  Organizational  Structure:  if  work  is  performed  differently  in  different  parts 
of  the  organizational  structure  (e.g.,  different  divisions  as  depicted  on  an 
organization  chart) 

•  Type  of  Work:  if  work  is  performed  differently  based  on  the  “type  of  work” 
(e.g.,  system  integration,  software  development,  IT-support  services,  or  help¬ 
desk) 

Each  of  these  potential  sampling  factors  shall  be  evaluated  for  its  effect  on  the 
conditions  under  which  work  is  performed  in  different  parts  of  the  organizational 
unit.  Other  sources  of  diversity  that  affect  these  conditions  shall  also  be 
considered  as  additional  potential  sampling  factors.  Examples  of  sampling 
factors  could  include  funding  source,  duration,  complexity,  or  other  factors  that 
affect  the  conditions  under  which  work  is  performed. 

Evaluations  of  potential  sampling  factors  shall  be  based  on  the  identification  of 
meaningful  values  (e.g.,  large  or  small  units;  the  name  of  cities  where  sites  are 
located)  into  which  basic  units  can  be  allocated  for  each  sampling  factor.  The 
number  of  subgroups  is  determined  by  analyzing  all  combinations  of  values 
within  the  sampling  factors.  The  result  is  a  set  of  subgroups  that  represents  the 
potentially  unique  conditions  under  which  work  is  perfonned  across  the 
organizational  unit. 

Potential  sampling  factors  that  do  not  drive  differences  in  conditions  under  which 
work  is  performed  may  reasonably  be  excluded.  Rationale  for  inclusion  or 
exclusion  of  these  factors  in  determining  a  representative  sample  shall  be 
documented  in  the  appraisal  plan,  along  with  the  subgroups  defined  by  the 
combination  of  all  relevant  sampling  factors. 
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1 .1 .4  Determine  Appraisal  Scope 


Parameters 
and  Limits 
(continued) 


Subgroups  defined  using  the  sampling  factors  determined  to  be  relevant  represent 
clusters  of  similar  conditions  for  process  implementation.  These  subgroups  contain 
one  or  more  basic  units  that  are  candidates  to  participate  in  the  appraisal  data 
collection  activities  (i.e.,  supply  artifacts  and/or  affirmations). 


Establish  a  representative  sample  for  the  organizational  unit  by  selecting  basic  units 
from  each  of  the  subgroups  according  to  the  following  fonnula: 


Minimum  mimtar 
of  basic  units  to 
b«  selected  from 
a  given  subgroup 


Number  of 
subgroups  X 


Number  of 
basic  units  In  the 
given  subgroup 


Total  number 
of  basic  units 


Figure  2:  Sampling  Formula 

The  computed  value  of  the  fonnula  above  may  generate  a  fractional  number.  If  the 
computed  value  using  this  formula  is  less  than  1,  then  the  required  number  of  basic 
units  shall  be  1.  Fractional  values  (greater  than  1)  resulting  from  the  computation 
shall  be  subject  to  standard  rounding  rules  (i.e.,  1.51  becomes  2,  and  1.49  becomes 
1).  The  organizational  scope  of  the  appraisal  is  defined  as  the  set  of  basic  units 
selected  using  the  process  described  above  and  the  identified  support  functions.  This 
set  of  organizational  entities  shall  provide  data  for  the  model  scope  of  the  appraisal, 
in  accordance  with  the  data  coverage  rules  specified  below. 

The  mapping  of  process  areas  to  basic  units  and  support  functions  shall  be  recorded 
in  order  to  establish  how  the  organizational  scope  of  the  appraisal  is  linked  to  the 
model  scope  of  the  appraisal.  Some  process  areas  may  be  instantiated  in  each  and 
every  basic  unit.  Other  process  areas  may  be  instantiated  in  a  single  support 
function,  ft  is  also  possible  that  a  given  process  area  is  instantiated  in  parallel 
support  functions  that  exist  within  given  subgroups,  or  are  shared  across  a  number 
of  subgroups. 

The  following  coverage  rules  ensure  corroboration  of  objective  evidence  from 
multiple  independent  sources  across  the  set  of  basic  units  or  support  functions 
sampled. 
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1.1.4  Determine  Appraisal  Scope 


Parameters 
and  Limits 
(continued) 


Coverage  Rules  for  Process  Areas 

Coverage-1:  Objective  evidence  provided  for  a  process  area  must  address  all 
practices  that  are  part  of  that  process  area.  For  CMMI  models,  this  includes  all 
specific  practices  as  well  as  all  generic  practices  included  in  the  scope  of  the 
process  area.  Each  basic  unit  or  support  function  sampled  must  address  all 
practices  in  the  process  areas  for  which  they  supply  data. 

Coverage-2:  In  terms  of  the  organizational  coverage  of  a  process  area,  the  design 
of  the  process  in  the  organization  may  lead  to  process  areas  implemented  by  one  of 
the  following: 

•  an  individual  basic  unit  within  a  subgroup 

•  a  single  support  function  that  serves  the  entire  organizational  unit 

•  a  set  of  support  functions  that  each  serve  different  parts  of  the  organizational 
unit 

•  some  hybrid  of  the  above  where  the  groupings  of  basic  units  in  subgroups 
would  be  too  limiting  (It  would  make  sense  to  ‘collapse  the  subgroups 
together’  for  the  purpose  of  looking  at  this  process  area  because  of  the  level  of 
standardization.) 

As  an  exception  to  the  following  coverage  rules  specified  for  basic  units  below,  the 
implementation  of  a  process  area  in  a  standardized  manner  across  subgroups  may 
reduce  the  data  collection  needs.  By  documenting  rationale  in  the  appraisal  plan, 
the  appraisal  team  leader  may  define  a  data  collection  strategy  that  collapses  the 
subgroups  when  considering  process  areas  implemented  in  a  highly  standardized 
fashion. 

Coverage  Rules  for  Basic  Units 

Coverage-1:  For  each  subgroup,  both  artifacts  and  affirmations  shall  be  provided 
for  at  least  one  basic  unit  for  every  process  area  implemented  by  basic  units  within 
that  subgroup.  This  sampled  basic  unit  shall  provide  data  for  all  process  areas. 
Selection  of  this  basic  unit  must  consider  the  schedule  of  work  (e.g.,  lifecycle 
stage)  achieved,  in  order  to  maximize  coverage  of  the  process  areas.  In  cases  where 
this  sampled  basic  unit  would  have  “not  yet”  characterizations  in  a  process  area, 
additional  basic  units  must  be  sampled  to  cover  that  process  area  (unless  no  other 
basic  units  remain  to  be  sampled  in  the  subgroup). 

Coverage-2:  For  at  least  50  percent  of  the  sampled  basic  units  in  each  subgroup, 
both  artifacts  and  affirmations  shall  be  provided  for  at  least  one  process  area 
implemented  by  basic  units  within  that  subgroup. 

Coverage-3:  For  all  sampled  basic  units  in  each  subgroup  either  artifacts  or 
affirmations  shall  be  provided  for  at  least  one  process  area  implemented  by  basic 
units  within  that  subgroup. 
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1.1.4  Determine  Appraisal  Scope 


Parameters 
and  Limits 
(continued) 


For  subgroups  with  only  one  sampled  basic  unit,  satisfaction  of  the  first  rule 
leads  to  satisfaction  of  the  other  two  rules. 

Coverage  Rules  for  Support  Functions 

Coverage-1:  Both  artifacts  and  affirmations  shall  be  provided  for  each  support 
function  for  all  process  areas  relating  to  the  work  performed  by  that  support 
function. 

Coverage-2:  The  artifacts  and  affirmations  provided  by  support  functions  shall 
demonstrate  the  work  performed  for  basic  units  for  at  least  one  sampled  basic 
unit  in  each  subgroup.  This  applies  for  each  process  area  relating  to  the  work 
performed  by  that  support  function  for  basic  units. 

For  example,  quality  assurance  or  configuration  management  functions  would 
provide  data  on  the  application  of  PPQA  or  CM  respectively.  The  linkage  of  the 
functions  to  assure  quality  or  maintain  configurations  of  key  work  products 
related  to  other  process  areas  does  not  mean  these  support  functions  must 
address  all  the  practices  in  those  other  process  areas. 

Coverage-3:  In  cases  where  multiple  support  functions  exist  within  the 
organizational  unit,  all  instances  of  the  support  function  shall  be  included  in  the 
appraisal  scope.  For  example,  if  division-specific  Configuration  Management 
groups  exist,  every  group  in  each  division  included  in  the  organizational  unit 
must  be  sampled. 

Organizational  unit  size  (i.e.,  number  of  people  and  number  of  basic  units)  and 
sizes  of  basic  units  (i.e.,  number  of  people)  in  the  organizational  scope  shall  be 
documented  in  the  appraisal  plan  as  well  as  the  percentage  ratio  of  these  two 
measures: 

•  Population  percent:  the  number  of  people  in  the  organizational  scope 
divided  by  the  number  of  people  in  the  organizational  unit  (xlOO) 

•  Basic  unit  percent:  the  number  of  basic  units  in  the  organizational  scope 
divided  by  the  number  of  basic  units  in  the  organizational  unit  (xlOO) 

Basic  units  that  are  specifically  excluded  from  participating  in  the  appraisal 
(though  they  are  included  in  the  definition  of  the  organizational  unit)  must  be 
identified  in  the  appraisal  plan  and  in  the  appraisal  disclosure  statement  along 
with  a  justification  for  their  exclusion.  Such  exclusions  might  arise  from 
logistical  constraints  identified  during  planning — such  as  unplanned  urgent 
customer  demands.  Units  that  are  not  part  of  the  organizational  unit  need  not  be 
listed  in  this  manner — as  the  results  do  not  apply  to  them. 
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1.1.4  Determine  Appraisal  Scope 


Parameters 
and  Limits 
(continued) 


Implementation 

Guidance 


The  appraisal  team  reserves  the  right  to  seek  clarification  or  data  from  other 
basic  units  or  support  functions  within  the  organizational  unit,  beyond  those 
specified  in  the  data  collection  plan.  These  basic  units  or  support  functions 
must  also  be  identified  in  the  appraisal  disclosure  statement. 

The  organizational  scope  to  be  investigated  during  the  appraisal  will  drive  the 
selection  of  participants  needed  to  provide  sources  of  objective  evidence.  An 
initial  determination  of  appraisal  participants,  by  name  and  role,  must  be 
negotiated  with  the  appraisal  sponsor  and/or  the  senior  site  manager  as  part  of 
the  early  determination  of  organizational  scope.  This  initial  determination  will 
be  refined  later  during  detailed  appraisal  planning. 

If  the  Conduct  Appraisal  phase  is  to  be  performed  using  incremental  subsets  of 
the  organizational  unit  or  the  model,  the  appraisal  plan  must  identify  the 
organizational  scope  and  appraisal  reference  model  scope  for  each  increment. 

Delta  appraisals  are  not  permitted.  A  delta  appraisal  is  defined  as  a  second 
appraisal  performed  on  a  subset  of  an  original  appraisal  model  scope  after 
correcting  weaknesses  reported  in  the  previous  appraisal,  and  then  combining 
the  results  of  the  second  appraisal  with  the  results  of  the  parts  of  the  first 
appraisal  that  were  not  investigated  in  the  second  appraisal  to  get  new  results. 


The  topic  of  sampling  and  determining  the  organizational  scope  can  be  a 
complex  issue,  with  considerable  impacts  on  the  credibility  of  the  appraisal 
overall  and  the  cost-effective  implementation  of  SCAMPI  A  appraisals.  A 
thorough  understanding  of  these  concepts  is  necessary  to  ensure  they  are 
consistently  and  practically  applied.  Appendix  F,  Scoping  and  Sampling  in 
SCAMPI  A  Appraisals,  provides  much  more  detailed  descriptions,  illustrations, 
and  case  studies  to  further  elaborate  these  concepts. 

The  selection  of  the  appraisal  reference  model  scope  should  have  been 
discussed  during  the  setting  of  appraisal  objectives.  Choices  regarding  the 
reference  model  and  the  selection  of  components  to  include  may  impact  the 
achievability  of  appraisal  objectives.  Clearly,  a  broadly-defined  organizational 
unit  (e.g.,  a  multi-national  enterprise)  will  require  collecting  and  analyzing 
significantly  more  objective  evidence  than  a  narrowly  defined  organizational 
unit  (e.g.,  a  specific  product  line  within  a  specific  business  unit  at  a  single 
geographical  location). 
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1.1.4  Determine  Appraisal  Scope 


Implementation 

Guidance 

(continued) 


The  organizational  unit  to  which  appraisal  results  will  be  attributed  should  be 
described  accurately  in  all  statements  made  by  the  appraisal  team  leader  and 
sponsor.  It  is  the  responsibility  of  the  appraisal  team  leader  to  understand  the 
larger  organizational  context  in  which  the  appraised  organizational  unit  resides. 
Familiarity  with  the  nature  of  departmental  structures,  matrixed  subject  matter 
expert  groups,  integrated  product  teams,  program  and  basic  unit  groupings,  or 
product  line  implications  that  may  affect  the  interpretation  of  appraisal 
outcomes  will  aid  in  obtaining  this  understanding. 


In  some  organizations,  a  small  number  of  process  areas  may  be  implemented 
uniformly  across  basic  units  to  an  extent  that  differences  across  subgroups  do 
not  exist  as  they  do  in  the  implementation  of  other  process  areas.  For  example, 
maintenance  projects  or  service  offerings  that  fall  under  a  common  program 
management  structure  may  be  governed  by  a  unified  set  of  work  processes 
relating  to  Project/Work  Monitoring  and  Control.  Such  a  program  (which  is  a 
collection  of  basic  units)  would  be  managed  by  common  management  lines 
with  shared  reporting  requirements. 


The  behavior  of  staff  in  the  program  is  guided  by  a  set  of  conditions  that 
override  the  influences  of  sampling  factors  that  lead  to  differences  in  other 
parts  of  the  organizational  unit.  In  these  exceptional  cases,  the  method  supports 
a  different  form  of  sampling  for  basic  units — one  that  considers  an  instantiation 
of  the  process  area  that  is  identical  among  basic  units  from  different  subgroups. 


The  organizational  unit  should  be  documented  in  the  clearest  terms  possible.  It 
is  often  difficult  to  specify  unambiguous  boundaries  without  resorting  to 
naming  individual  people  in  some  organizations.  Information  about  the 
organizational  unit  should  be  documented  in  a  way  that  allows  future  appraisal 
sponsors  to  replicate  (to  the  extent  possible)  the  organizational  unit  appraised. 
This  information  should  be  in  the  appraisal  plan,  and  used  (in  summary  form  if 
needed)  in  briefing  the  appraisal  team  and  appraisal  participants. 


♦ 
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1.1.5  Determine  Appraisal  Outputs 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Identify  the  specific  appraisal  outputs  to  be  produced.  Some  appraisal  outputs  are 
required  and  additional  outputs  are  tailorable  (see  Parameters  and  Limits).  Some 
possible  SCAMPI  outputs  are  supported  by  all  reference  models;  other  possible 
outputs  are  reference-model  specific  (see  model-specific  implementation 
guidance  below). 

Obtain  information  to  answer  the  following  questions: 

•  What  ratings  will  be  generated  during  the  appraisal? 

•  Will  a  final  report  be  written  to  document  appraisal  results? 

•  Will  recommendations  on  how  to  address  specific  findings  be  generated  and 
reported? 


The  appraisal  team  leader  or  designee  shall 

•  review  required  SCAMPI  A  outputs  with  the  appraisal  sponsor 

•  review  and  select  optional  SCAMPI  A  outputs  with  the  appraisal  sponsor 


Required  SCAMPI  A  outputs  include 

•  appraisal  record  (see  activity  3.2.2,  Generate  Appraisal  Record) 

•  SEI  data  (see  activity  3.2.3,  Provide  Appraisal  Feedback  to  the  SEI) 

At  least  all  the  goals  for  the  process  area  or  process  areas  within  the  model  scope 
must  be  rated  for  the  organizational  unit,  although  the  choice  may  be  made  to  not 
disclose  the  ratings  to  anyone  other  than  the  appraisal  sponsor.  Ratings  for 
individual  disciplines  or  for  individual  basic  units,  unless  the  basic  unit  is  the 
organizational  unit,  are  not  allowed. 

The  sponsor  shall  receive  the  appraisal  record,  which  includes 

•  final  findings,  including  statements  of  strengths  and  weaknesses 

•  documented  for  every  process  area  investigated 

•  all  ratings  planned  for  and  generated  by  the  team 

•  the  appraisal  disclosure  statement 

The  appraisal  team  leader  and  sponsor  are  required  to  sign  the  appraisal 
disclosure  statement. 
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1.1.5  Determine  Appraisal  Outputs 


Implementation 

Guidance 


Non-model  findings  reflect  items  that  have  significant  positive  or  negative 
impact  on  the  enactment  of  processes  within  the  organizational  unit  that  do  not 
directly  relate  to  model  practices. 

While  statements  of  findings  (strengths  and  weaknesses)  are  a  required  output 
of  the  method,  creating  a  written  report  that  elaborates  on  the  findings  is 
optional.  The  sponsor  should  decide  if  resources  are  to  be  spent  creating  this 
artifact. 

Similarly,  the  task  of  creating  recommendations  to  address  appraisal  results 
may  require  expertise  that  is  not  represented  on  the  appraisal  team  in  some 
cases.  Recommendations  may  address  suggestions  to  resolve  a  weakness  or  to 
propagate  a  strength.  When  requested,  recommendations  are  often  included  in 
final  findings;  however,  the  sponsor  may  request  a  separate  recommendations 
report  that  elaborates  on  the  recommendations. 

Additionally,  the  characteristics  of  the  appraised  organization  and  the 
constraints  that  shape  its  improvement  program  should  be  carefully  considered 
when  generating  a  process  improvement  action  plan  that  defines  the  tasks 
necessary  to  address  weaknesses. 


♦ 
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1.1.6  Obtain  Commitment  to  Initial  Appraisal  Plan 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  appraisal  sponsor  formally  approves  the  initial  appraisal  plan,  and  this  set 
of  information  is  placed  under  change  management. 


The  appraisal  team  leader  or  designee  shall 

•  record  required  information  in  the  initial  appraisal  plan 

•  obtain  sponsor  approval  of  the  initial  appraisal  plan 

•  manage  changes  to  the  initial  appraisal  plan,  obtaining  sponsor  approval 
of  changes 


The  appraisal  plan  is  often  generated  incrementally  throughout  the  Plan  and 
Prepare  for  Appraisal  phase,  and  must  be  approved  prior  to  the  start  of  the 
conduct  appraisal  phase.  The  initial  appraisal  plan  addresses  the  requirements 
of  the  appraisal,  and  guides  future  appraisal  planning.  At  a  minimum,  the 
initial  appraisal  plan  provide  the  information  needed  to  address  the  following: 

•  the  identity  of  the  appraisal  sponsor  and  the  relationship  of  the  sponsor  to 
the  organizational  unit  being  appraised 

•  the  appraisal  purpose,  including  alignment  with  business  objectives  (see 
activity  1.1.1,  Determine  Appraisal  Objectives) 

•  the  organizational  unit  being  appraised 

•  the  appraisal  scope  (see  activity  1.1.4,  Determine  Appraisal  Scope) 

•  the  organizational  scope  of  the  appraisal 

•  the  reference  model  scope 

•  sampling  factors  affecting  the  appraisal 

•  the  process  context,  which  includes,  at  a  minimum 

organizational  unit  size  and  demographics 
application  domain,  size,  criticality,  and  complexity 
high-priority  characteristics  (e.g.,  time  to  market,  feature  richness, 
and  reliability)  of  the  products  and  services  of  the  organizational  unit 

•  the  data  collection  strategy  (see  activity  1.1.2,  Determine  Data  Collection 
Strategy)  which  include 

the  data  collection  approach  (discovery,  managed  discovery,  and/or 
verification) 

the  data  collection  timing  (e.g.,  preparation  phase  or  conduct  phase) 
the  data  collection  techniques  (e.g.,  demonstrations,  presentations, 
interviews  and  questionnaires) 
responsibility  for  data  collection 
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1.1.6  Obtain  Commitment  to  Initial  Appraisal  Plan 


Parameters 
and  Limits 
(continued) 


•  the  appraisal  constraints  (see  activity  1.1.3,  Determine  Appraisal 
Constraints)  which  include  at  a  minimum 

availability  of  key  resources  (e.g.,  staffing,  funding,  tools,  and 
facilities) 

schedule  constraints 

the  maximum  amount  of  time  to  be  used  for  the  appraisal  (The 
maximum  time  to  perform  the  Conduct  Appraisal  phase  is  90  days.) 
specific  process  areas  or  organizational  entities  to  be  excluded  from 
the  appraisal 

the  maximum,  minimum,  or  specific  sample  size  or  coverage  desired 
for  the  appraisal 

ownership  of  appraisal  results  and  any  restrictions  on  their  use 
controls  on  information  resulting  from  a  confidentiality  agreement 
non-attribution  of  appraisal  outputs  to  individuals 

•  the  identity  of  the  appraisal  reference  models  used  (version,  addition,  and 
representation) 

•  any  process  area  designated  as  “not  applicable”  and  the  rationale  for  its 
exclusion  (note  that  “not  applicable”  areas  may  preclude  ratings,  for  some 
reference  models 

•  the  rationale  for  selecting  the  sample  basic  units  and  support  functions  as 
representative  of  the  organizational  unit 

•  basic  units,  categories,  or  groups/functions  that  are  specifically  excluded 
from  the  appraisal  as  well  as  the  justification  for  their  exclusion 


CMU/SEI-2011-HB-001  |  57 


1.1.6  Obtain  Commitment  to  Initial  Appraisal  Plan 


Parameters 
and  Limits 
(continued) 


Implementation 

Guidance 


•  the  identity  and  affiliation  of  the  SCAMPI  Lead  Appraiser  who  is  to  be 
the  appraisal  team  leader  for  the  appraisal 

•  the  identity  and  affiliation  of  the  appraisal  team  members  and  their 
specific  appraisal  responsibilities 

•  the  identity  (i.e.,  name  and  organizational  affiliation)  of  appraisal 
participants  and  support  staff,  and  their  specific  responsibilities  for  the 
appraisal 

•  any  additional  information  to  be  collected  during  the  appraisal  to  support 
the  achievement  of  the  appraisal  objectives 

•  a  description  of  the  planned  appraisal  outputs  (see  activity  1.1.5, 
Determine  Appraisal  Outputs),  including  ratings  to  be  generated 

•  anticipated  follow-on  activities  (e.g.,  reports,  appraisal  action  plans,  or  re¬ 
appraisal) 

•  planned  tailoring  of  SCAMPI  A  and  associated  tradeoffs 

•  appraisal  usage  mode  (i.e.,  Internal  Process  Improvement,  Supplier 
Selection,  or  Process  Monitoring) 


An  appraisal  team  leader’s  ability  to  build  and  maintain  commitment  from  the 
sponsor  and  the  members  of  the  sponsoring  organization  is  a  major  factor 
contributing  to  the  success  of  the  appraisal.  The  process  of  understanding  the 
requirements  and  constraints  should  yield  a  series  of  agreements  that  form  an 
input  to  the  appraisal  plan.  Based  on  the  judgment  of  the  appraisal  team  leader, 
these  agreements  may  be  covered  in  a  formal  (signed)  document  that  forms  a 
basis  for  future  activities.  More  typically,  the  appraisal  team  leader  maintains  a 
record  of  interactions  with  the  sponsor,  which  are  incorporated  into  the 
appraisal  plan  as  it  is  drafted. 

The  appraisal  team  leader  and  the  sponsor  should  have  verbal  agreement  on 
the  items  discussed  above,  and  these  items  should  be  documented  in  some 
way.  The  formality  of  the  documentation  may  range  from  simple  meeting 
minutes  maintained  by  the  appraisal  team  leader,  to  a  more  formal 
memorandum  of  understanding  or  other  vehicle  that  documents  agreements 
and  provides  traceability.  It  is  expected  that  the  appraisal  plan  will  be  used  to 
document  important  issues  pertaining  to  requirements. 


♦ 
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1.2  Develop  Appraisal  Plan 


Purpose 

Document  the  results  of  appraisal  planning  including  the  requirements, 
agreements,  estimates,  risks,  method  tailoring,  and  practical  considerations 
(e.g.,  schedules,  logistics,  and  contextual  information  about  the  organization) 
associated  with  the  appraisal.  Obtain  and  record  the  sponsor’s  approval  of  the 
appraisal  plan. 

Entry  Criteria 

The  appraisal  sponsor  and  appraisal  team  leader  have  agreed  to  proceed  with 
appraisal  planning  based  on  a  common  understanding  of  the  key  parameters 
that  drive  the  planning  process. 

Inputs 

Documented  agreements,  reflected  in  the  appraisal  plan,  that  support  a 
common  understanding  of  appraisal  objectives  and  key  appraisal-planning 
parameters 

Activities 

1.2.1  Tailor  Method 

1 .2.2  Identify  Needed  Resources 

1 .2.3  Develop  Data  Collection  Plan 

1.2.4  Determine  Cost  and  Schedule 

1 .2.5  Plan  and  Manage  Logistics 

1 .2.6  Document  and  Manage  Risks 

1 .2.7  Obtain  Commitment  to  Appraisal  Plan 

Outputs 

•  approved  appraisal  plan 

Outcome 

The  sponsor  and  appraisal  team  leader  agree  on  technical  and  non-technical 
details  for  the  planned  appraisal.  The  plan  is  refined  in  conjunction  with 
performing  the  other  Planning  and  Preparation  phase  activities.  This 
agreement  is  documented  and  reviewed  by  affected  stakeholders  as 
appropriate. 

Exit  Criteria 

The  final  appraisal  plan  is  reviewed  and  approved. 
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1.2  Develop  Appraisal  Plan 


Key  Points 

Skilled  appraisal  team  leaders  will  effectively  develop  and  use  outputs  from 
the  other  Planning  and  Preparation  phase  activities  to  achieve  clarity  of  the 
shared  vision  necessary  to  make  the  tradeoffs  and  decisions  resulting  in  a  final 
plan.  This  activity  is  an  important  opportunity  for  the  appraisal  team  leader  to 
demonstrate  process  discipline,  as  well  as  the  type  of  careful  planning 
described  in  the  reference  models.  Experienced  appraisal  team  leaders  will 
leverage  data,  templates,  and  assets  (developed  through  their  own  experience) 
to  improve  the  completeness  and  effectiveness  of  the  appraisal  plan, 
recognizing  the  return  on  investment  that  will  be  obtained  through  smooth  and 
efficient  appraisals. 

Tools  and 
Techniques 

Tools  include  an  appraisal  plan  template,  samples,  and  embedded  procedural 
guidance  in  planning  templates.  Estimation  worksheets  and  methods  for 
assessing  the  impact  of  appraisal  constraints  are  also  quite  useful. 

Metrics 

•  calendar  time  spanned  by  the  activity 

•  effort  consumed  in  carrying  out  the  activities  of  this  process 

•  level  and  frequency  of  changes  to  the  appraisal  plan 

Verification  and 
Validation 

•  comparison  of  actual  effort  for  this  activity  with  historical  data 
accumulated  by  the  appraisal  team  leader 

•  review  of  the  appraisal  plan  by  affected  stakeholders 

•  sponsor’s  approval  of  the  plan 

Records 

•  estimation  worksheets  (if  used) 

•  appraisal  plan  (see  activity  1.2.7,  Obtain  Commitment  to  Appraisal  Plan, 
for  a  detailed  list  of  plan  contents) 
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1.2  Develop  Appraisal  Plan 


Interfaces 
with  Other 
Processes 


The  appraisal  plan  will  guide  and  define  the  execution  of  the  appraisal  such  that 
it  is  in  concert  with  the  business  needs  and  constraints.  An  initial  plan  can  be 
generated  immediately  following  consultation  with  the  sponsor.  Further 
refinement  is  done  as  detailed  planning  occurs  and  new  information  comes  to 
light  in  executing  appraisal  planning  and  preparation.  A  final  appraisal  plan 
must  be  completed  prior  to  the  completion  of  process  1.5,  Prepare  for  Appraisal 
Conduct.  Typically,  resources,  method  tailoring,  model-related  decisions,  and  a 
planned  list  of  outputs  are  finalized  early  on,  while  cost,  schedule,  and  logistics 
are  finalized  later  in  the  Plan  and  Prepare  for  Appraisal  phase. 

While  it  may  not  be  necessary  to  formally  separate  the  requirements  analysis 
activities  from  the  activities  described  in  this  section,  prior  understanding  of  the 
appraisal  requirements  is  a  necessary  input  to  this  process.  The  plan  for  the 
appraisal  provides  an  important  vehicle  for 

•  documenting  agreements  and  assumptions 

•  establishing  and  maintaining  sponsorship 

•  tracking  and  reporting  the  performance  of  the  appraisal  process 

•  reinforcing  commitments  at  key  points  in  the  appraisal  process 


The  initial  version  of  the  appraisal  plan  is  intended  to  capture  key  appraisal 
requirements  and  strategic  objectives,  which  require  high  sponsor  visibility  and 
change  control  approval.  Later  versions  of  the  appraisal  plan  add  the  tactical 
planning  details  necessary  to  implement  and  satisfy  these  objectives. 


Summary  of 
Activities 


This  process  is  composed  of  the  activities  summarized  here  and  described 
below.  The  scope  of  the  appraisal  is  defined  in  terms  of  (a)  the  portion  of  the 
appraisal  reference  model  that  will  be  investigated  and  (b)  the  bounds  of  the 
organizational  unit  for  which  the  results  can  be  considered  valid  (e.g.,  a  basic 
unit,  a  product  line,  a  work  group,  an  operating  division,  a  business  unit,  or  an 
entire  global  enterprise).  Method-tailoring  choices  are  made  to  most  effectively 
achieve  appraisal  objectives  within  defined  constraints  of  time,  effort,  and  cost. 
The  resources  required  to  carry  out  the  appraisal  are  identified.  The  cost  and 
schedule  are  negotiated  and  recorded.  The  details  of  logistics,  particularly  for 
the  Conduct  Appraisal  phase,  are  documented.  Risks  and  risk-mitigation  plans 
are  identified  and  documented.  Completion  of  these  activities  results  in  a  well- 
defined,  achievable  appraisal  plan. 


♦ 
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1.2.1  Tailor  Method 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Tailoring  of  SCAMPI  A  includes 

•  selection  of  choices  (if  any)  within  the  Required  Practices 

•  setting  parameters  that  are  allowed  to  vary  within  the  Parameters  and 
Limits 

Because  SCAMPI  A  is  designed  to  apply  to  a  wide  range  of  appraisal 
applications,  the  tailoring  activity  is  one  that  deserves  careful  and  thoughtful 
attention. 

The  SCAMPI  MDD  is  designed  to  clearly  indicate  which  aspects  of  the  method 
are  required  and  which  are  tailorable.  The  Parameters  and  Limits  sections  of 
each  activity  description  provide  discussions  of  tailoring  options,  in  context. 

In  addition,  the  appraisal  usage  mode  will  determine  some  tailoring  choices. 


The  appraisal  team  leader  or  designee  shall 

•  review  and  select  tailoring  options  within  each  activity 

•  ensure  that  the  tailoring  decisions  are  self-consistent  and  that  they  are 
appropriate  in  light  of  the  appraisal  objectives  and  constraints 

•  document  the  tailoring  decisions  made  in  the  appraisal  plan 


The  structure  of  the  MDD  clarifies  which  SCAMPI  A  features  are  required. 
Parameters  and  Limits  sections  define  the  allowable  variation  within  these 
method  requirements.  Tailoring  guidance  and  Implementation  Guidance  are 
provided  to  assist  with  tuning  the  method  to  fit  sponsor  objectives  and  appraisal 
constraints. 
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1.2.1  Tailor  Method 


Implementation 

Guidance 


This  appraisal  method  offers  a  wide  variety  of  choices  that  allow  the  appraisal 
team  leader  and  sponsor  to  select  appraisal  features  that  best  address  appraisal 
and  business  objectives. 

Method  tailoring  is  directly  related  to  the  organizational  scope  and  appraisal 
reference  model  scope  decisions.  Most  of  the  allowable  tailoring  options  flow 
logically  from  these  decisions  when  taken  in  context  of  the  appraisal  objectives 
and  constraints.  Tailoring  decisions  typically  affect  the  appraisal  risk.  Typical 
tailoring  choices  that  significantly  impact  appraisal  planning  include 

•  assigning  mini-teams  by  basic  unit  or  by  process  area  grouping 

•  data  collection  approaches  to  be  utilized  (discovery,  managed  discovery 
and/or  verification)  and  the  associated  supporting  work  aids  and  tools  (e.g., 
use  of  video  conference,  teleconference,  or  other  similar  technology  to 
conduct  interviews,  conducting  parallel  interview  sessions  with  a  minimum 
of  two  team  members,  and  use  of  a  database  of  objective  evidence  mapped 
to  model  practices) 

•  verification  approaches  to  be  utilized,  including  supporting  work  aids  and 
tools  (e.g.,  mini  team  verification  of  practices  at  the  instantiation  level) 

•  validation  approaches  to  be  utilized  including  supporting  work  aids  and 
tools  (e.g.,  use  of  instrument  or  targeted  focus  group  for  validation  of 
preliminary  findings) 

•  selection  of  optional  SCAMPI  outputs  (e.g.,  preliminary  findings  focused 
on  basic  units,  division,  or  disciplines,  and  the  goal  ratings,  capability  level 
ratings,  and  maturity  level  ratings) 

•  documenting  non-model  findings 

•  optional  activities  (e.g.,  conduct  executive  session,  plan  for  next  steps,  or 
collect  lessons  learned) 

Experienced  appraisal  team  leaders  will  provide  a  well-defined  approach  to 
ensure  that  the  appraisal  objectives  are  achieved  in  an  efficient  and  effective 
manner.  Experienced  sponsors  will  require  a  well-defined  approach  to  ensure 
an  acceptable  level  of  risk  in  meeting  objectives  within  the  constraints.  The 
appraisal  plan  documents  the  method-tailoring  decisions  and  their  rationale, 
and  the  associated  method  variations  and  techniques  that  will  be  employed. 


♦ 
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1.2.2  Identify  Needed  Resources 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


This  activity  is  concerned  with  the  identification  and  estimation  of  resources 
needed  to  carry  out  the  appraisal.  Resources  include  personnel,  facilities,  tools, 
and  access  to  information. 


The  appraisal  team  leader  or  designee  shall  do  the  following: 

•  identify  appraisal  team  members 

•  identify  appraisal  participants 

•  identify  equipment  and  facilities 

•  identify  other  appraisal  resources  needed 

•  document  resource  decisions  in  the  appraisal  plan 


The  level  of  detail  in  the  identification  of  needed  resources  must  be  sufficient 
to  support  the  creation  of  the  appraisal  plan.  At  a  minimum,  the  appraisal  team 
leader  must  identify  the  following: 

•  the  names  of  people  who  are  candidates  for  affirmations  or  appraisal  team 
membership,  and  support  personnel 

•  the  organizational  or  basic  unit  affiliation  of  these  people 

•  the  location,  seating  capacity,  and  configuration  of  rooms  to  be  used  by 
the  team 

•  specific  equipment  needed  (e.g.,  overhead  projector,  laptop  projector,  or 
video  -conferencing) 
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1.2.2  Identify  Needed  Resources 


Implementation 

Guidance 


Appraisal  resources  are  typically  defined  early  in  the  appraisal-planning 
process.  Identifying  resources  goes  hand  in  hand  with  estimating  appraisal  cost 
and  schedule  (see  activity  1.2.4,  Determine  Cost  and  Schedule),  and  these 
resources  may  be  iteratively  refined.  Tradeoffs  are  routinely  made  in  light  of 
the  appraisal  objectives  and  constraints. 

The  appraisal  sponsor  or  senior  site  manager  may  identify  candidate  appraisal 
team  members  and  appraisal  participants.  Review  of  the  organizational  unit 
structure  or  other  site-specific  information  can  also  be  useful  for  this 
identification.  Initially,  participants  can  be  specified  in  terms  of  roles  or 
responsibilities,  with  specific  names  to  be  determined  later.  Process  1.3,  Select 
and  Prepare  Team,  contains  additional  guidance  on  selecting  appraisal  team 
members. 

Equipment  and  facilities  are  often  negotiated  with  the  organizational  unit  where 
the  appraisal  activities  will  be  performed,  but  sometimes  these  equipment  and 
facilities  must  be  acquired.  A  room  for  dedicated  use  by  the  appraisal  team  is 
usually  necessary  for  private  discussions  and  to  protect  the  confidentiality  of 
appraisal  data.  Ideally,  this  room  is  separate  from  the  other  rooms  where 
interview  sessions  are  held. 

The  availability  of  computing  resources,  such  as  computers,  printers,  and 
networks,  is  a  key  consideration  that  should  be  planned  and  understood.  Access 
to  special  tools  or  applications  may  also  be  needed. 


♦ 


CMU/SEI-2011-HB-001  |  65 


1.2.3  Develop  Data  Collection  Plan 


Activity 

Description 

The  data  collection  plan  is  based  on  the  data  collection  strategy  defined  in  activity 
1.1.2,  Determine  Data  Collection  Strategy.  It  is  considered  a  part  of  the  overall 
appraisal  plan  and  is  part  of  the  record  submitted  at  the  end  of  the  appraisal.  It  is 
typically  documented  as  section  of  the  appraisal  plan.  However,  it  can  also  be 
documented  as  an  appendix  to  the  appraisal  plan  or  as  a  standalone  document. 

This  plan  information  is  higher  level  information  that  helps  document  and 
communicate  the  overall  approach  to  data  collection  for  the  appraisal. 

The  data  collection  activities  are  tailored  to  meet  the  needs  for  objective  evidence 
so  that  the  extent  of  practice  implementation  can  be  determined. 

For  practices  that  have  objective  evidence,  a  strategy  for  verifying  that  evidence 
will  be  formulated. 

For  practices  that  lack  objective  evidence,  a  strategy  for  discovering  that  evidence 
will  be  formulated. 

Detailed  information  on  data  collection  can  be  recorded  in  work  aids  that  manage 
appraisal  data  and  in  the  appraisal  schedule.  A  record  of  “information  needed” 
items  is  the  most  detailed,  while  artifact  lists,  interview  schedules,  and  the 
assigmnent  of  process  area  mini-teams  help  shape  the  strategy  for  obtaining  the 
needed  data.  The  documented  data  collection  plan  will  explicitly  state  where  this 
detailed  information  will  be  recorded.  The  data  collection  plan  will  evolve  and  be 
revised  as  the  appraisal  is  planned  and  performed.  Analysis  of  the  data  collection 
plan  can  be  used  to  identify  important  risks  to  the  appraisal  which  will  be 
documented  in  the  risk  section  of  the  appraisal  plan. 

Required 

Practices 

The  appraisal  team  leader  or  designee  shall  document  the  data  collection  plan. 

Parameters 
and  Limits 

The  data  collection  plan  must  specify  contingencies  to  manage  the  risk  of  having 
insufficient  data. 

For  every  instantiation  of  every  model  practice,  the  data  collection  plan  must 
specify  how,  when,  and  by  whom  the  objective  evidence  will  be  verified. 

For  instantiations  of  model  practices  that  have  not  been  addressed  in  the  initial 
objective  evidence,  the  data  collection  plan  must  specify  how  the  team  intends  to 
discover  the  presence  or  absence  of  objective  evidence  that  characterizes  the 
extent  of  implementation  for  that  practice. 
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1.2.3  Develop  Data  Collection  Plan 


Parameters 
and  Limits 
(continued) 


The  data  collection  plan  is  considered  part  of  the  appraisal  plan  and  is  often 

documented  in  a  variety  of  artifacts,  which  may  be  completed  at  different  phases 

of  appraisal  planning.  The  data  collection  plan  includes 

•  documentation  of  the  data  collection  strategy  (see  activity  1.1.2,  Determine 
Data  Collection  Strategy) 

•  identification  of  participants  to  be  involved  in  affirmation  activities 

•  assignment  of  process  areas  to  team  members 

•  the  schedule  and  success  criteria  for  readiness  reviews 

•  the  approach  for  using  Class  C  and  Class  B  appraisals  for  data  collection 
and/or  readiness  reviews  (if  they  are  used  for  this  purpose) 

•  a  summary  of  initial  objective  evidence  provided  by  the  organization 

•  identification  of  highest  priority  data  needs 

•  a  schedule  of  interview-gathering  events,  revised  over  time  to  include  more 
detail 

•  for  interviews,  the  identification  of  an  initial  set  questions  (this  may  be 
maintained  outside  of  the  overall  appraisal  plan) 

•  identification  of  artifacts  still  needed  after  the  performance  of  readiness 
reviews  (if  any) 

•  risks  associated  with  the  sufficiency  of  the  data  and  the  adequacy  of  the 
schedule 
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1.2.3  Develop  Data  Collection  Plan 


Implementation 

Guidance 


Types  of  objective  evidence  include  artifacts  and  affirmations  (see  process  2.2, 
Examine  Objective  Evidence).  A  combination  of  these  evidence  types  is 
required  for  corroboration  (see  activity  1.1.4,  Determine  Appraisal  Scope).  The 
data  collection  status  is  continually  monitored  during  appraisal  activities  (see 
process  2.3,  Document  Objective  Evidence)  to  ensure  that  sufficient  data 
coverage  is  obtained.  These  key  considerations  should  be  understood  and 
accounted  for  in  the  generation  of  the  data  collection  plan. 

Multiple  types  of  interviews  can  be  used  to  obtain  affirmations  (see  activity 
2.2.2,  Examine  Objective  Evidence  from  Affirmations): 

•  standard  structured  interviews  scheduled  in  advance  that  use  scripted 
questions 

•  on-call  interviews,  scheduled  in  advance  for  calendar  purposes,  but  held 
only  if  it  is  determined  they  are  necessary 

•  office  hours  interviews  in  which  interviewees  are  notified  that  they  may 
need  to  be  available  as  a  contingency  during  scheduled  periods 

The  data  collection  plan  should  specify  sources  of  data,  tools,  and  techniques  to 
be  used. 

A  robust  data  collection  plan  will  plan  for  interviews  of  all  three  types.  Start 
with  a  frill  set  of  scheduled  interviews  and  gradually  add,  eliminate,  or  modify 
events  as  the  inventory  of  initial  objective  evidence  indicates  the  need. 

The  data  collection  plan  should  clearly  specify  whether  any  virtual  methods 
(e.g.,  video  conferences,  teleconferences,  and  other  similar  technology)  will  be 
used  and  how  they  will  be  used.  It  is  the  responsibility  of  the  appraisal  team 
leader  to  ensure  that  virtual  methods  do  not  compromise  the  integrity  or 
accuracy  of  appraisal  activities  or  appraisal  results. 

Planning  for  evidence  reviews  should  include  artifacts  as  described  in  activity 
2.2.1,  Examine  Objective  Evidence  from  Artifacts. 

Ultimately,  the  appraisal  team  must  have  data  on  each  practice  in  the  reference 
model  within  the  appraisal  scope  for  each  organizational  element  within  the 
appraisal  scope.  For  process  areas  addressing  practices  implemented  at  the  basic 
unit  level  (e.g.,  Project  Planning),  this  requirement  means  that  data  will  be 
collected  for  each  practice  from  each  basic  unit  selected  to  provide  data  for  that 
process  area.  For  process  areas  addressing  practices  implemented  at  the 
organization  level  (e.g.,  Organizational  Training),  only  one  instantiation  of  each 
practice  may  be  needed,  depending  on  the  way  the  organization  chooses  to 
implement  such  practices. 
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1.2.3  Develop  Data  Collection  Plan 


Implementation 

Guidance 

(continued) 


The  results  of  the  analysis  of  initial  objective  evidence  are  used  to  determine 
which  practices  are  not  already  covered  with  objective  evidence.  Practices  for 
which  no  initial  objective  evidence  has  been  provided  should  be  identified  as 
high-risk  areas  for  the  team  to  address  immediately.  The  schedule  for  data 
collection  may  need  to  change  dramatically  if  the  team  is  unable  to  find 
relevant  data  for  these  areas  in  short  order.  In  the  case  of  practices  for  which 
data  are  available  in  the  initial  objective  evidence,  the  team  members 
assigned  to  the  process  areas  plan  the  strategy  for  verifying  the 
implementation  of  each  of  the  practices  through  review  of  the  named 
artifacts,  affirmations  from  the  people  who  fill  the  named  roles,  or  other  data 
collection  events.  Artifacts  used  to  manage  data  collection  events  are 
populated  with  the  current  understanding  of  the  planned  data  collection 
events,  as  follows: 


•  The  schedule  for  interviews  is  finalized,  so  participants  can  be  informed 
of  the  expectations  for  their  participation  as  interviewees. 

•  The  list  of  documents  on  hand  (e.g.,  accessible  electronically)  is  finalized 
so  that  the  team  members  know  what  is  and  is  not  available  for  document 
review. 

•  A  preliminary  allocation  of  practices  to  be  covered  in  each  of  the 
scheduled  interviews  is  documented. 

A  list  of  needed  documents  (not  yet  available  to  the  team)  is  generated,  if 

there  are  any  known  needs  for  documents  at  this  point. 


♦ 
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1.2.4  Determine  Cost  and  Schedule 


Activity 

Description 

A  top-level  cost  breakdown  and  schedule  are  developed  and  included  in  the 
plan. 

Required 

Practices 

The  appraisal  team  leader  or  designee  shall 

•  estimate  the  duration  of  key  events  as  a  basis  for  deriving  a  comprehensive 
schedule 

•  estimate  the  effort  required  for  the  people  participating  in  the  appraisal 

•  estimate  the  costs  associated  with  using  facilities  and  equipment  as 
appropriate 

•  estimate  the  costs  for  incidentals  (e.g.,  travel,  lodging,  and  meals)  as 
appropriate 

•  document  a  detailed  schedule  in  the  appraisal  plan 

•  document  detailed  cost  estimates  in  the  appraisal  plan 

Parameters 
and  Limits 

Scheduling  for  each  day  of  the  appraisal  is  required. 
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1.2.4  Determine  Cost  and  Schedule 


Implementation 

Guidance 


Cost  and  schedule  may  be  developed  top-down  based  on  sponsor  objectives 
and  constraints,  bottom  up  based  on  results  of  other  planning  and  preparation 
processes  and  activities,  or  more  generally  using  a  combination  of  the  two 
approaches.  Scheduling  the  events  and  activities  of  the  appraisal  is  an  ongoing 
logistical  task  that  requires  the  coordination  of  many  different  groups  of 
individuals.  Determining  and  communicating  a  schedule  for  the  appraisal,  and 
maintaining  ongoing  visibility  as  the  details  take  form,  is  the  primary 
responsibility  of  the  appraisal  team  leader.  The  appraisal  coordinator  is 
expected  to  provide  support  in  this  task,  and  the  appraisal  team  leader  typically 
selects  the  person  who  plays  that  role  with  this  duty  in  mind. 

The  needs  of  the  sponsor  for  appraisal  outputs  of  a  specified  quality  fulfilling  a 
specified  purpose,  balanced  against  the  resources  available  to  conduct  the 
appraisal,  will  determine  the  schedule  constraints.  Schedule  and  cost  must  be 
considered  for  the  entire  span  of  the  appraisal  activities.  Effort  estimates 
should  be  developed  not  only  for  the  appraisal  team,  but  also  for  the  expected 
participants  within  the  organizational  unit  (e.g.,  interviewees,  respondents  to 
instruments  administered,  attendees  at  briefings,  and  support  staff). 

Organizational  costs  for  preparing  and  supporting  appraisals  can  be  reduced 
by  gathering  and  maintaining  objective  evidence  for  each  instantiation.  In 
addition  to  providing  an  effective  mechanism  for  monitoring  the  process 
implementation  and  improvement  progress  of  the  organizational  unit,  this 
approach  enables  the  ready  availability  and  reuse  of  objective  evidence  for 
subsequent  appraisals. 

While  the  schedule  for  the  appraisal  is  shared  with  a  fairly  wide  audience,  the 
cost  of  the  appraisal  (or  basic  units  within  the  appraisal)  is  often  kept  from 
wide  view,  due  to  the  potentially  sensitive  nature  of  this  information. 


♦ 
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1.2.5  Plan  and  Manage  Logistics 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Implementation 

Guidance 


The  logistical  details  of  the  appraisal  are  negotiated  and  documented.  The 
appraisal  team  leader,  supported  by  the  appraisal  coordinator,  manages  planning 
tasks  that  document  and  communicate  logistical  arrangements.  Checklists  and 
action  item  tracking  mechanisms  are  important  structures  used  to  manage  these 
tasks. 


The  appraisal  team  leader  or  designee  shall 

•  document  logistical  schedules  and  dependencies 

•  maintain  communication  channels  for  providing  status 

•  assign  responsibilities  for  tracking  logistical  issues 


None 


Effective  planning  depends  on  anticipating  a  variety  of  logistical  issues  that 
may  occur  during  the  appraisal.  The  time-critical  nature  of  appraisal  activities 
makes  it  difficult  to  manage  last-minute  changes  in  important  details  such  as  the 
following: 

•  identifying  hotels  for  people  traveling  to  the  appraisal 

•  providing  transportation  and/or  lodging  for  team  members  or  the  remote 
members  of  the  organizational  unit 

•  providing  workstation  support 

•  ordering  meals 

•  interacting  with  facilities  staff  on  site 

•  meeting  security/classification  requirements 

•  providing  badges  or  arranging  for  escorts  in  limited-access  facilities 

•  providing  access  to  rooms,  equipment,  and  supplies  needed  for 
administrative  tasks 

•  providing  use  of  virtual  methods  (e.g.,  video  conferences,  teleconferences, 
and  other  similar  technology)  to  conduct  appraisal  activities 

•  providing  communication  channels  and  back-up  staff  to  support  the  team 
on  site 

If  virtual  methods  such  as  video  conferences,  teleconferences,  and  other  similar 
technology  are  to  be  used  to  perform  appraisal  activities,  these  methods  should 
be  clearly  defined  in  the  Appraisal  Plan.  Furthermore,  it  is  the  responsibility  of 
the  appraisal  team  leader  to  ensure  that  the  use  of  virtual  methods  in  no  way 
compromises  the  integrity  or  accuracy  of  the  appraisal  activities  or  the  appraisal 
results.  Virtual  methods  should  allow  for  adequate  interaction  between  the 
appraisal  team  members  and  the  appraisal  participants  and  should  provide 
mechanisms  for  the  appraisal  team  to  control  the  interactions. 


♦ 
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1.2.6  Document  and  Manage  Risks 


Activity 

Description 

As  with  any  activity  containing  dependencies  among  events,  people,  and  other 
resources,  risk  management  is  an  important  ingredient  to  success.  The  appraisal 
team  leader  is  responsible  for  documenting  and  communicating  risks  and 
associated  mitigation  plans  to  the  sponsor  and  appraisal  team  members. 

Required 

Practices 

The  appraisal  team  leader  or  designee  shall 

•  identify  appraisal  risks 

•  develop  mitigation  plans  for  key  appraisal  risks  and  implement  these  plans  as 
necessary 

•  keep  the  appraisal  sponsor  and  other  stakeholders  informed  of  the  appraisal 
risk  status 

The  risks  and  mitigation  plans  identified  through  conducting  this  activity  are 
required  elements  of  the  appraisal  plan  (see  Parameters  and  Limits  for  activity 

1.2.7,  Obtain  Commitment  to  Appraisal  Plan).  If  an  identified  risk  occurs  during 
appraisal  execution  then  this  should  also  be  documented  in  the  plan. 

The  appraisal  team  leader  is  responsible  for  keeping  the  appraisal  sponsor 
informed  of  risk  management  activities  so  that,  if  needed,  timely  sponsor 
intervention  is  possible  to  ensure  the  achievement  of  appraisal  objectives. 

Parameters 
and  Limits 

When  evaluating  risks  to  an  appraisal  the  following  potential  risk  areas  must  be 
considered: 

•  personnel 

•  logistic 

•  technical 

•  facilities 

•  schedule 

CMU/SEI-2011-HB-001  73 


1.2.6  Document  and  Manage  Risks 


Implementation 

Guidance 


Risk  management  is  the  systematic  process  of  planning  for,  identifying, 
analyzing,  responding  to,  and  monitoring  risks.  It  involves  processes,  tools,  and 
techniques  that  help  the  appraisal  team  leader  maximize  the  probability  and 
results  of  positive  events  and  minimize  the  probability  and  consequences  of 
adverse  events  as  indicated  and  appropriate  within  the  context  of  risk  to  the 
overall  appraisal  objectives.  Risk  management  is  most  effective  when 
performed  early  in  the  appraisal  planning  process  and  is  a  continuing 
responsibility  of  the  appraisal  team  leader  throughout  the  appraisal. 

Most  appraisal  team  leaders  include  a  section  titled  “Risk  Management”  in  the 
appraisal  plan.  The  level  of  effort  devoted  to  risk-management  activities  is 
something  the  appraisal  team  leader  must  adjust  to  fit  the  situation. 


♦ 
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1 .2.7  Obtain  Commitment  to  Appraisal  Plan 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Formal  sponsor  commitment  to  the  appraisal  plan  is  obtained.  The  appraisal 
plan  constitutes  a  contract  between  the  appraisal  sponsor  and  the  appraisal  team 
leader,  so  it  is  vital  that  this  agreement  be  formal. 


The  appraisal  team  leader  or  designee  shall 

•  document  the  appraisal  plan 

•  review  the  appraisal  plan  with  the  sponsor  and  secure  the  sponsor’s 
approval 

•  provide  the  appraisal  plan  to  relevant  stakeholders  for  review 


Required  contents  of  the  appraisal  plan  include  the  following,  at  a  minimum: 

•  the  initial  appraisal  plan  (see  activity  1.1.6,  Obtain  Commitment  to  Initial 
Appraisal  Plan) 

•  the  activities  to  be  performed  in  conducting  the  appraisal 

•  resources  needed  for  conducting  the  appraisal  (see  activity  1.2.2,  Identify 
Needed  Resources) 

•  data  collection  plan  (see  activity  1 .2.3,  Develop  Data  Collection  Plan) 

•  cost  and  schedule  estimates  for  performing  the  appraisal  (see  activity  1.2.4, 
Determine  Cost  and  Schedule) 

•  appraisal  logistics  (see  activity  1.2.5,  Plan  and  Manage  Logistics) 

•  risks  and  mitigation  plans  associated  with  appraisal  execution  (see  activity 
1.2.6,  Document  and  Manage  Risks) 

There  must  be  a  signature  block  for  the  appraisal  team  leader  and  the  sponsor  to 
indicate  in  writing  their  commitment  to  the  plan.  If  minor  updates  are  made  to 
the  plan,  signatures  do  not  have  to  be  obtained  again.  If  changes  affect  the 
scope  (model  or  organizational)  of  the  appraisal,  then  the  plan  must  be  re¬ 
baselined. 

At  a  minimum,  the  appraisal  team  members  are  considered  relevant 
stakeholders  and  should  receive  a  copy  of  the  approved  appraisal  plan. 


♦ 
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1.3  Select  and  Prepare  Team 


Purpose 

Ensure  that  an  experienced,  objective,  trained,  and  appropriately  qualified  team  is 
available  and  prepared  to  execute  the  appraisal  process. 

Entry  Criteria 

•  Appraisal  requirements  have  been  documented  (at  least  in  draft  form). 

•  Appraisal  constraints  are  understood  and  documented  (at  least  in  draft  form). 

•  The  appraisal  plan  is  defined  (at  least  in  draft  form). 

Inputs 

•  appraisal  requirements  and  constraints  (in  draft  or  final  form) 

•  appraisal  plan  (in  draft  form) 

•  team  training  materials 

Activities 

1.3.1  Identify  Appraisal  Team  Leader 

1.3.2  Select  Team  Members 

1.3.3  Document  and  Manage  Conflicts  of  Interest 

1.3.4  Prepare  Team 

Outputs 

•  training  records 

•  appraisal  team  member  assigmnents  and  qualifications 

•  identified  and  documented  conflicts  of  interest 

•  a  prepared  appraisal  team  that  has  completed 

appraisal  method  training 
appraisal  reference  model  training 
team-building  activities 
team  orientation  regarding  appraisal 

Outcome 

The  successful  completion  of  this  process  results  in  an  experienced,  objective, 
qualified,  and  trained  team  ready  to  execute  the  appraisal.  The  appraisal  team 
members  have  acquired  the  necessary  knowledge  to  play  their  roles,  or  their 
previous  knowledge  is  confirmed  to  be  satisfactory.  The  appraisal  team  leader 
has  provided  opportunities  to  practice  the  skills  needed  for  each  person  to  play 
his/her  role,  or  has  confirmed  that  these  skills  have  already  been  demonstrated  in 
the  past.  The  team  members  have  been  introduced  to  one  another,  and  have 
begun  to  plan  how  they  will  work  together. 

Exit  Criteria 

•  The  prepared  team  is  committed  to  the  appraisal. 

•  Training  has  been  provided  and  its  results  recorded. 

•  Conflicts  of  Interest  have  been  mitigated. 

•  Remediation  of  knowledge/ skill  shortfalls  has  been  completed  (if  needed). 
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1 .3  Select  and  Prepare  Team 


Key  Points 


Tools  and 
Techniques 


Metrics 


Verification 
and  Validation 


Records 


Whether  the  appraisal  team  leader  trains  an  intact  team  or  fonns  a  team  from  a 
corps  of  experienced  team  members,  the  responsibility  to  ensure  that  the  team  is 
ready  to  succeed  rests  with  the  appraisal  team  leader. 


Training  course  material  is  available  from  the  SEI  for  training  teams.  This 
training  should  be  tailored  or  supplemented  by  the  appraisal  team  leader  based  on 
the  appraisal  context  or  degree  of  team  member  experience.  Case  studies  and 
exercises  are  recommended  to  reinforce  the  situations  team  members  are  likely  to 
encounter  during  the  appraisal. 

Other  ways  of  accomplishing  this  activity  may  draw  on  one  or  more  of  the 
following: 

•  providing  supplementary  training  to  previously  experienced  team  members 
so  that  the  operational  details  of  the  approach  used  will  be  familiar 

•  training  a  cadre  of  team  members  and  keeping  their  knowledge  and  skills  up- 
to-date  as  part  of  an  overall  program  of  appraisals 


•  summary  of  team  member  qualifications 

•  effort  and  calendar  time  expended  to  accomplish  training 

•  trainee  ratings  of  instructional  materials  and  approach  (if  applicable) 

•  achievement  of  milestones  for  remedial  activities  (if  applicable) 


•  sponsor  and  appraisal  team  leader  approval  of  identification  and  mitigation 
of  conflicts  of  interest,  team  membership  and  preparation 

•  results  of  exams  used  to  demonstrate  training  effectiveness  (if  used) 

•  feedback  from  team  members  on  their  readiness  to  perform  their  role(s) 


•  team  member  contact  information 

•  training  records  (if  applicable) 

•  feedback  provided  by  trainees  (if  applicable) 

•  team  qualification  summary  (recorded  in  appraisal  plan) 
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1 .3  Select  and  Prepare  Team 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


This  process  includes  selecting  appraisal  team  members,  identifying  and 
mitigating  conflicts  of  interest,  and  preparing  the  appraisal  team.  It  may  occur 
after  obtaining  sponsor  commitment  to  the  initial  appraisal  plan.  The  appraisal 
plan  should  be  available,  at  least  in  draft  form,  as  a  necessary  input  (see  activity 
1.2.7,  Obtain  Commitment  to  Appraisal  Plan  for  contents).  Selected  appraisal 
team  members  may  provide  input  into  further  definition  of  the  appraisal 
planning.  Appraisal  team  training  may  provide  an  initial  means  to  obtain  a 
preliminary  understanding  of  the  appraised  organization’s  operations  and 
processes.  If  available,  the  organizational  unit’s  database  of  objective  evidence 
mapped  to  model  practices  is  a  useful  resource  for  orienting  the  appraisal  team 
on  organizational  characteristics,  such  as  the  application  domain,  the 
organizational  structure,  the  process  improvement  structure,  and  approaches  for 
appraisal  reference  model  implementation. 


The  appraisal  team  is  a  cohesive  unit  of  objective,  trained  and  capable 
professionals,  each  of  whom  must  meet  stringent  qualifications.  An  appraisal 
team  leader  is  selected  to  plan  and  manage  the  performance  of  the  appraisal, 
delegate  appraisal  tasks  to  team  members,  and  ensure  adherence  to  SCAMPI  A 
requirements.  Appraisal  team  members  are  selected  based  on  defined  criteria 
for  experience,  objectivity,  knowledge,  and  skills  to  ensure  an  efficient  team 
capable  of  satisfying  the  appraisal  objectives.  Training  is  provided  to  ensure 
proficiency  in  the  appraisal  reference  model  and  appraisal  method. 


♦ 
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1.3.1  Identify  Appraisal  Team  Leader 


Activity 

Description 


Required 

Practices 


Parameters  and 
Limits 


The  appraisal  sponsor  selects  an  appraisal  team  leader  who  has  the  appropriate 
experience,  knowledge,  and  skills  to  take  responsibility  for  and  lead  the 
appraisal.  By  definition,  an  appraisal  team  leader  is  a  SCAMPI  Lead 
Appraiser,  certified  by  the  SEI  Appraisal  Program,  and  is  a  member  of  that 
program  in  good  standing.  The  SEI  Appraisal  Program  is  described  on  the  SEI 
web  site:  <http://www.sei.cmu.edu/>.  The  appraisal  team  leader  ensures  that 
the  appraisal  is  conducted  in  accordance  with  SCAMPI  A  requirements,  with 
tailoring  to  meet  appraisal  objectives  and  constraints  within  allowable  bounds 
defined  by  the  method. 


The  sponsor  or  designee  shall 

•  select  a  certified  SCAMPI  Lead  Appraiser  to  serve  as  the  appraisal  team 
leader 

•  verify  the  qualifications  of  the  appraisal  team  leader  (experience, 
knowledge,  and  skills) 


The  appraisal  team  leader  must  be  an  SEI-certified  SCAMPI  Lead  Appraiser 
in  good  standing  (or  a  candidate  SCAMPI  Lead  Appraiser  or  People  CMM 
Appraiser  being  observed  by  a  qualified  observing  lead  appraiser).  This 
certification  can  be  verified  on  the  web  or  by  contacting  the  SEI  directly. 

If  the  SCAMPI  A  to  be  performed  includes  the  rating  of  high  maturity  process 
areas,  the  appraisal  team  lead  must  be  an  SEI-certified  SCAMPI  high-maturity 
lead  appraiser  in  good  standing.  This  certification  can  be  verified  on  the  web, 
or  by  contacting  the  SEI  directly. 

There  can  be  only  one  official  appraisal  team  leader  on  any  given  appraisal. 
The  appraisal  team  leader  has  sole  discretion  to  delegate  important  tasks  to 
appraisal  team  members,  but  cannot  delegate  leadership  responsibility  or 
ultimate  responsibility  for  the  successful  completion  of  the  appraisal.  The 
inclusion  of  multiple  SCAMPI  Lead  Appraisers  on  a  team  for  a  given 
appraisal  can  be  a  strong  asset  for  the  leader  of  that  team.  However,  the  single 
designated  appraisal  team  leader  must  perfonn  the  leadership  role  and  manage 
the  appraisal  process. 
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1.3.1  Identify  Appraisal  Team  Leader 


Implementation 

Guidance 


SCAMPI  Lead  Appraisers,  by  definition,  will  have  participated  on  a  minimum 
of  three  appraisals  (two  as  an  appraisal  team  member  and  one  as  an  appraisal 
team  leader).  These  requirements  are  outlined  in  the  SEI  Lead  Appraiser 
program.  An  additional  consideration  impacting  team  experience  requirements, 
however,  is  the  appraisal  usage  mode  for  SCAMPI  A.  Additional  experience 
may  be  necessary  if  the  appraisal  is  for  supplier  selection  and/or  process 
monitoring,  or  if  it  will  focus  on  other  disciplines  or  environments.  Similarly,  if 
the  appraisal  will  be  used  in  a  high  maturity  organization  (maturity  levels  4-5), 
additional  team  qualifications  are  needed  as  listed  in  activity  1.3.2,  Select  Team 
Members. 

Appraisal  team  leader  responsibilities  are  defined  and  described  throughout  the 
SCAMPI  A  MDD,  but  a  summary  overview  of  these  responsibilities  includes 
the  following: 

•  Confirm  the  sponsor’s  commitment  to  proceed  with  the  appraisal. 

•  Ensure  that  appraisal  participants  are  briefed  on  the  purpose,  scope,  and 
approach  of  the  appraisal. 

•  Ensure  that  all  appraisal  team  members  have  the  appropriate  experience, 
knowledge,  and  skills  in  the  appraisal  reference  model  and  in  SCAMPI  A. 

•  Ensure  that  the  appraisal  is  conducted  in  accordance  with  the  documented 
SCAMPI  A  method. 

•  Verify  and  document  that  the  appraisal  method  requirements  have  been 
met. 

The  appraisal  team  leader  may  be  selected  at  any  time  in  the  appraisal  planning 
phase;  preferably,  the  appraisal  team  leader  is  selected  upon  initiation  of 
appraisal  activities  so  that  he  or  she  may  participate  in  analyzing  the 
requirements  with  the  appraisal  sponsor.  In  any  event,  the  appraisal  team  leader 
is  identified  in  time  to  (a)  review  and  approve  the  appraisal  plan  with  the 
appraisal  sponsor  prior  to  beginning  the  Conduct  Appraisal  phase  of  the 
appraisal,  and  (b)  ensure  adequate  planning  and  the  preparation  of  appraisal 
team  members. 


♦ 
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1.3.2  Select  Team  Members 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


This  activity  involves  identifying  available  personnel,  assessing  their 
qualifications,  and  selecting  them  to  become  appraisal  team  members.  It  may 
occur  after  obtaining  the  sponsor’s  commitment  to  conduct  the  appraisal  and  may 
provide  input  to  appraisal  planning. 


The  appraisal  team  leader  shall 

•  select  individual  team  members  that  meet  the  minimum  criteria  for 
individual  team  members 

•  select  individual  team  members  that  collectively  meet  the  minimum  criteria 
for  the  team  as  a  whole 

•  document  the  qualifications  and  responsibilities  of  team  members  in  the 
appraisal  plan. 


The  minimum  acceptable  team  size  for  a  SCAMPI  A  appraisal  is  four  people 
(including  the  appraisal  team  leader). 

Each  Appraisal  Team  Member  must  have  previously  completed  an  SEI-licensed 
offering  of  the  designated  introductory  course  relating  to  each  and  every 
reference  model  (e.g.,  CMMI  Constellation,  or  People  CMM)  that  is  included  in 
the  scope  of  the  appraisal. 

The  team  overall  must  have  field  experience  relating  to  the  content  of  the 
reference  model  (e.g.,  development  of  products  and  services  for  CMMI-DEV, 
delivery  of  services  for  CMMI-SVC,  acquisition  for  CMMI-ACQ,  human 
resources  for  P-CMM).  For  each  reference  model  (e.g.,  constellation)  in  the 
scope  of  the  appraisal,  the  team  must  have  the  following: 

•  Individual  team  members,  each  of  whom  have  at  least  two  years  of 
experience  performing  the  type  of  work  addressed  in  each  appraisal 
reference  model  included.  This  means  each  team  member  has  field 
experience  with  each  included  model.  The  appraisal  team  leader,  at  their 
discretion,  may  accept  one  team  member  with  no  field  experience  - 
however,  this  must  be  documented  along  with  the  rationale  for  this  exception 
in  the  appraisal  plan. 

•  An  average  of  at  least  six  years  of  field  experience  excluding  the  experience 
of  the  appraisal  team  leader,  relating  to  the  content  of  each  of  the  reference 
models.  This  ensures  that  team  members  with  relatively  limited  experience 
are  balanced  by  others  who  have  a  greater  level  of  experience. 

•  An  aggregate  of  25  years  of  field  experience  relating  to  the  content  of  each 
of  the  reference  models  excluding  the  experience  of  the  appraisal  team 
leader.  This  ensures  that  the  team  as  a  whole  has  a  substantial  experience 
base  to  support  their  judgments. 
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1.3.2  Select  Team  Members 


Parameters 
and  Limits 
(continued) 


The  appraisal  team  leader  is  required  to  evaluate  and  validate  the  team  members 
experience  by  either  a  review  of  the  team  members  resume  or  interview  each 
team  member  to  determine  their  level  of  expertise. 

The  team  (as  a  group)  must  have  a  total  of  at  least  ten  years  of  management 
experience,  and  at  least  one  team  member  must  have  at  least  six  years  of 
experience  as  a  manager  -  excluding  the  appraisal  team  leader. 

The  selected  appraisal  team  members  and  their  organizational  affiliation  and 
qualifications  (individually  and  in  aggregate)  must  be  documented  in  the 
appraisal  plan.  The  team  leader  must  identify  the  range  of  roles,  functions  or 
activities  performed  in  the  work  done  within  the  organizational  unit  (e.g., 
lifecycle  stages  for  CMMI-DEV). 

The  appraisal  team,  as  a  whole,  must  have  members  with  experience  performing 
practices  from  all  of  the  process  areas  included  in  the  appraisal  scope.  The 
appraisal  team  shall  not  be  comprised  entirely  of  staff  who  wrote  the  processes 
being  appraised.  If  one  or  more  process  authors  are  included  on  the  team,  the  risk 
management  section  of  the  appraisal  plan  must  address  how  potential  conflicts  of 
interest  will  be  managed. 

The  sponsor  of  the  appraisal  shall  not  be  an  appraisal  team  member.  A  senior 
manager  who  has  supervisory  authority  over  the  entire  organizational  unit  shall 
not  be  an  appraisal  team  member. 

Additional  requirements  specific  to  High  Maturity  appraisals: 

•  Ensure  all  members  of  the  high  maturity  mini-team  have  high  maturity 
experience. 

•  A  high  maturity  lead  appraiser  or  appraisal  team  member  with  statistical 
analysis  and  other  high  maturity-related  training  and  experience  shall  be 
assigned  to  all  mini-teams  focused  on  high  maturity  process  areas. 


The  team  as  a  whole  must  have  collective  experience  implementing  high 
maturity  activities  such  as  establishing,  evaluating,  using,  or  analyzing  process 
performance  baselines  and  process  performance  models. 

The  appraisal  team  leader  is  the  final  authority  on  acceptance  of  appraisal  team 
members  and  is  responsible  for  ensuring  their  qualifications  and  suitability  for 
the  appraisal  purpose. 
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1.3.2  Select  Team  Members 


Implementation 

Guidance 


Although  not  required  in  the  Parameters  and  Limits  section,  the  following  are 
considered  recommended  best  practices  and  should  be  employed  whenever 
feasible: 


•  Each  member  should  have  good  written  and  oral  communication  skills,  the 
ability  to  facilitate  the  free  flow  of  communication,  and  the  ability  to  perform 
as  team  players  and  negotiate  consensus. 

•  At  least  half  of  the  team  members  should  have  participated  in  a  previous 
process  appraisal. 

•  Team  members  should  be  perceived  by  the  appraisal  sponsor  as  credible. 


Additional  appraisal  team  member  selection  considerations  include 

•  Consider  the  personal  characteristics  of  individual  team  members  (e.g., 
communication  preferences  and  personality  types)  and  how  these 
characteristics  may  affect  the  dynamics  of  the  team. 

•  Using  more  internal  or  external  team  members  based  on  appraisal  parameters 
like  model  scope  or  organization  size 

•  Recruiting  team  members  from  other  divisions,  or  external  organizations 
where  applicable  and  appropriate 

•  Encourage  the  sponsor  or  internal  process  lead  or  appraisal  coordinator  to 
identify  team  members  that  are  external  to  the  organizational  unit  or 
organization  in  order  to  get  a  greater  component  of  non-organizational  unit 
members 

•  In  small  organizations,  where  the  appraised  organizational  unit  may  equal 
the  full  organization,  it  may  be  more  appropriate  to  select  some  external 
personnel  (outside  of  the  organization)  as  appraisal  team  members 
(cooperation  with  other  small  organizations  to  exchange  team  members  is 
also  an  option).  For  large  organizations,  persons  external  to  the 
organizational  unit,  e.g.  from  another  corporate  function,  may  bring  in  the 
same  objectivity  as  an  external  person.  All  these  situations  should  be 
considered,  discussed  with  the  sponsor  and  rationale  should  be  given  in  the 
appraisal  plan. 

•  Making  tradeoffs  between  team  size  and  team  member  expertise  levels 

•  Appraisal  constraints,  such  as  security  classification,  may  be  additional 
criteria  for  team  member  selection. 

•  Continuity  of  team  members  across  multiple  appraisals 
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1.3.2  Select  Team  Members 


Implementation 

Guidance 

(continued) 


•  Encourage  involvement  of  team  members  who  participated  on  previous 
appraisals  within  the  organization,  in  order  to  benefit  from  legacy 
experience. 

•  Consider  lessons  learned  from  prior  events  in  adjusting  the  team  for  the 
current  event 

•  Consider  bringing  in  some  new  team  members  to  get  a  fresh  perspective 

•  For  a  series  of  appraisals  (e.g.  for  a  SCAMPI  C,  B,  A  sequence  or  related 
OU)  strive  for  maintaining  overall  team  member  continuity.  Consider 
exchanging  team  members  by  assigning  them  to  other  tasks.  Another  option 
might  be  to  add  team  members  while  moving  from  C  to  B  to  A  (also  due  to 
changing  minimum  team  size). 

•  Making  tradeoffs  between  assigning  specialized  roles  to  certain  team 
members  versus  a  ‘more  uniform  distribution  of  tasks.’ 

•  If  the  appraisal  team  leader  allows  mini  teams  to  ask  the  questions  during  an 
interview,  ensure  that  the  mini-teams  are  aware  of  the  rules  associated  with 
interviews. 

•  Consider  adjusting  the  work  load  of  team  members  that  are  assigned 
specialized  responsibilities  such  as  time  keeper,  etc.  as  the  situation  dictates. 

•  Use  one  or  more  certified  SCAMPI  Lead  Appraisers  as  team  members. 

The  maximum  recommended  team  size  is  nine,  but  a  balance  between  the  scope 
of  the  appraisal  and  the  size  of  the  team  should  be  considered.  Team  member 
training  in  the  appraisal  method  is  discussed  in  activity  1.3.4,  Prepare  Team. 

Team  members  should  not  be  managers  of  any  of  the  selected  basic  units  or 
support  functions  or  be  within  the  direct  supervisory  chain  of  any  of  the 
anticipated  interviewees. 

Appraisal  team  members  are  selected  to  provide  a  diverse  set  of  qualified 
professionals  with  the  appropriate  experience,  knowledge,  and  skills  to  make 
reasoned  judgments  regarding  implementation  of  the  appraisal  reference  model. 
The  accuracy  and  credibility  of  the  appraisal  results  depends  greatly  on  the 
capability,  qualifications,  and  preparation  of  the  appraisal  team  members.  In 
addition  to  the  qualifications  described  above,  other  factors  that  may  affect  the 
performance  of  the  team  or  reliability  of  appraisal  results  should  be  considered. 
Appraisal  constraints,  such  as  security  classification,  may  be  additional  criteria 
for  team  member  selection. 
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1.3.2  Select  Team  Members 


Implementatio 
n  Guidance 
(continued) 


Appraisal  team  members  are  typically  selected  from  a  pool  of  qualified 
individuals  provided  by  the  appraisal  sponsor  or  his/her  designee.  Situations 
where  a  conflict  of  interest  may  arise  should  be  avoided.  Team  members  who 
manage  people  or  processes  in  the  organization  may  struggle  with  their  ability  to 
be  objective.  Team  members  who  are  directly  impacted  by  the  appraisal  outcome 
may  be  distracted  by  the  potential  consequences  of  the  decisions  they  contribute 
to  on  the  appraisal  team. 


Additional  guidance  specific  to  High  Maturity  appraisals: 


•  Consider  having  a  certified  high  maturity  lead  appraiser  lead  the  high 
maturity  mini-team. 

•  Consider  splitting  up  high  maturity  skills  on  the  team  onto  multiple  mini¬ 
teams  to  spread  input  and  balance  experience  across  the  process  areas  in 
scope.  Assign  team  members  with  particular  knowledge  (e.g.  statistical 
techniques),  to  different  mini-  teams  to  balance  expertise  among  teams. 


♦ 
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1.3.3  Document  and  Manage  Conflicts  of  Interest 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


This  activity  involves  identifying  and  handling  conflicts  of  interest  that  may 
impair  an  appraisal  team’s  ability  to  function  objectively.  The  appraisal  team 
leader  is  responsible  for  handling  potential  conflicts  of  interest  by  avoiding  or 
developing  strategies  to  manage  them. 


The  appraisal  team  leader  shall 

•  identify  potential  conflicts  of  interest 

•  take  steps  to  avoid  those  conflicts  of  interest  that  can  be  avoided 

•  develop  strategies  to  manage  potential  conflicts  of  interest  that  cannot  be 
avoided,  and  document  those  strategies  in  the  appraisal  plan 

•  monitor  the  conflicts  of  interest  to  ensure  that  management  strategies  are 
effective 

•  take  appropriate  corrective  action  when  conflict  of  interest  management 
strategies  do  not  work 


The  appraisal  team  leader  must  use  professional  judgment  and  evaluate,  at  a 

minimum,  the  following  potential  conflicts  of  interest,  review  them  with  the 

sponsor  and  ensure  detailed  documentation  and  rationale  in  their  appraisal  plan: 

•  Are  any  team  members  authors  of  one  or  more  processes  included  in  the 
scope  of  the  appraisal? 

•  Are  any  members  of  the  organization’s  process  group  serving  on  the 
appraisal  team? 

•  Are  any  “process  owners”  serving  as  appraisal  team  members? 

•  Are  staff  with  supervisory  responsibility  over  one  or  more  parts  of  the 
organizational  unit  on  the  team? 

•  Are  people  who  served  on  previous  appraisals  (Class  C,  B  or  A)  of  the 
organizational  unit  serving  as  appraisal  team  members? 

•  Are  any  of  the  appraisal  team  members  in  a  direct  reporting  relationship 
above  any  appraisal  participants  or  other  appraisal  team  members,  including 
administrative,  functional,  basic  unit,  performance,  or  technical  authority 
(e.g.,  supervisory,  basic  unit,  program,  technical)? 

•  Will  any  of  the  appraisal  team  members  be  interviewed  or  providing 
evidence? 

•  Are  members  of  the  appraisal  team  involved  in  process  or  product  quality 
audits? 
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1.3.3  Document  and  Manage  Conflicts  of  Interest 


Parameters 
and  Limits 
(continued) 


If  using  a  translator, 

Is  the  translator  an  appraisal  team  member? 

Does  the  translator  work  for  the  appraised  company? 

Does  the  translator  work  within  the  organizational  unit? 

In  support  of  this  appraisal,  was  the  appraisal  team  leader  or  any  of  the 
appraisal  team  members  providing  coaching,  consulting,  or  labor  for  the 
organizational  unit's  project  work  or  process  identification, 
documentation,  or  creation?  (Providing  SEI-licensed  training  is 
excluded.) 


The  sponsor  of  the  appraisal  shall  not  be  an  appraisal  team  member  (see  activity 
1.3.2,  Select  Team  Members). 

Potential  conflicts  of  interest  that  cannot  be  avoided,  along  with  the  strategy  to  be 
used  to  manage  them,  are  to  be  documented  in  the  risk  management  section  of 
the  appraisal  plan. 

The  appraisal  team  leader  must  keep  the  appraisal  sponsor  informed  of 
management  activities  for  conflicts  of  interest  so  that,  if  needed,  timely  sponsor 
intervention  is  possible  to  ensure  the  achievement  of  appraisal  objectives. 

Caution  must  be  exercised  to  ensure  that  confidentiality  and  non-attribution 
requirements  of  the  appraisal  are  maintained. 

In  the  event  that  conflicts  of  interest  prove  unmanageable,  or  compromise  the 
team’s  objectivity  and  ability  to  reach  valid  conclusions,  the  appraisal  team  lead 
must  take  appropriate  steps,  up  to  and  including  termination  of  the  appraisal. 

The  level  of  effort  devoted  to  conflict  of  interest  management  activities  is 
something  the  appraisal  team  leader  must  adjust  to  fit  the  existing  situation. 
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1.3.3  Document  and  Manage  Conflicts  of  Interest 


Implementation 

Guidance 


The  following  practices  to  manage  potential  conflicts  of  interest  should  be 

considered  whenever  possible: 

Pairing  of  internal  and  external  team  members  on  a  mini-team 

•  Pairing  internal  and  external  team  members  provides  benefits  of  external 
objectivity  and  internal  knowledge. 

•  If  mini-teams  are  planned,  ensure  each  mini-team  has  at  least  one  member 
with  experience  in  the  process  areas  to  which  they  are  assigned 

Balancing  mini-teams  based  on  expertise  or  areas  of  interest 

•  If  this  is  the  first  SCAMPI  A  for  any  of  the  appraisal  team  members,  ensure 
they  are  paired  with  an  experienced  appraisal  team  member  and/or  give 
them  experience  on  other  types  of  appraisal  events  prior  to  the  A. 

•  If  any  mini-teams  do  not  have  at  least  one  member  with  experience  from  at 
least  one  SCAMPI  A  then  consider  the  following: 

appraisal  team  leader  spends  more  time  with  that  mini-team  during  the 
event 

include  that  mini-team  on  other  appraisal  events  prior  to  the  A  to 
increase  their  experience  level 

provide  a  greater  amount  of  training  prior  to  the  appraisal 
assign  the  process  areas  with  the  least  risk  to  the  mini-team 
potentially  increase  the  team  size  or  extend  duration  to  account  for 
lack  of  appraisal  experience 

•  If  all  appraisal  team  members  are  not  experienced  in  all  product 
development,  service  delivery,  and/or  project  management  areas  in  scope: 

pair  mini-teams  based  on  known  experience  to  mitigate  areas  where 
some  have  less  experience  in  a  particular  area 

Identify  internal  and  external  team  members 

•  External  team  members  do  not  work  for  the  organizational  unit,  but  may  be 
from  the  same  organization. 

•  Internal  team  members  work  directly  for  the  organizational  unit  being 
appraised. 

•  Document  how  many  appraisal  team  members  will  be  external  to  the 
organizational  unit,  including  the  appraisal  team  leader. 

•  Document  the  number  of  internal  appraisal  team  members. 

•  Document  the  number  of  external  appraisal  team  members  that  are  from  the 
organization,  but  not  the  organizational  unit. 

•  If  any  appraisal  team  members  perform  process  evaluations  for  the 
organizational  unit,  assign  them  to  mini-teams  covering  process  areas 
unrelated  to  those  evaluations. 

•  Do  not  assign  an  appraisal  team  member  to  evaluate  a  process  they 
improved  or  developed. 


♦ 
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1.3.4  Prepare  Team 


Activity 

Description 


Required 

Practices 


The  appraisal  team  leader  ensures  that  appraisal  team  members  are  sufficiently 
prepared  for  performing  the  planned  appraisal  activities.  This  preparation 
includes  ensuring  team  members  are  familiar  with  the  appraisal  reference  model, 
the  appraisal  method,  the  appraisal  plan,  organizational  data  and  characteristics, 
and  the  tools  and  techniques  to  be  used  during  the  appraisal.  Roles  and 
responsibilities  are  assigned  for  appraisal  tasks.  Team  building  exercises  are  used 
to  practice  facilitation  skills  and  reach  unity  in  understanding  the  team  objectives 
and  how  they  will  be  satisfied. 

All  team  members  are  expected  to  observe  strict  mles  for  confidentiality,  the 
protection  of  proprietary  or  sensitive  data,  and  the  non-attribution  of  information 
to  appraisal  participants.  Non-disclosure  statements  are  often  used  to  formalize 
these  understandings. 


The  appraisal  team  leader  shall 

•  ensure  that  appraisal  team  members  have  received  appraisal  reference  model 
training 

•  provide  appraisal  method  training  to  appraisal  team  members  or  ensure  that 
they  have  already  received  it 

•  foster  team  building  and  establishing  team  norms 

•  provide  an  orientation  to  team  members  on  appraisal  objectives,  plans,  and 
their  assigned  roles  and  responsibilities 

•  ensure  that  no  appraisal  activity  (performed  by  team  members)  begins  until 
after  the  method  training  relating  to  that  activity  has  been  completed. 
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1.3.4  Prepare  Team 


Parameters 
and  Limits 


Model  training  must  be  provided  using  the  standard  introductory  course  for  the 
model(s)  in  scope,  delivered  by  an  instructor  who  is  certified  by  the  SEI. 

At  a  minimum,  all  team  members  must  be  trained  on  the  following  topics  using 
information  from  the  SCAMPI  A  team  training  materials  provided  by  the  SEI: 

•  SCAMPI  A  method  overview 

•  appraisal  planning,  including  the  contents  of  the  appraisal  plan 

•  objective  evidence  collection  and  analysis 

•  team  decision  making 

•  appraisal  confidentiality  and  non-attribution 

•  practice  characterization 

•  findings  development,  verification,  and  validation 

•  rating 

•  appraisal  output  requirements 

For  teams  involved  in  U.S.  government  source  selection  or  process  monitoring 
appraisals,  team  members  must  also  be  trained  in 

•  applicable  laws,  regulations,  and  policies  that  affect  the  appraisal  such  as 
Federal  Acquisition  Regulations  and  DoD  service  or  organizational 
regulations  and  policies 

•  role  of  the  appraisal  and  the  appraisal  team  in  source  selection  or  process 
monitoring  processes  and  structures 

•  limitations  on  findings  development,  validation,  and  release 

•  special  domain  and/or  model  requirements  (e.g.  space,  command  and 
control,  information  technology,  supplier  sourcing,  and  statistical  process 
management) 

Appraisal  activities  may  not  be  performed  until  the  method  training  for  those 
activities  has  been  provided. 

Three  configurations  of  method  training  are  recognized: 

•  to  a  single  appraisal  team 

•  to  multiple  appraisal  teams  in  a  single  event 

•  to  a  large  group  of  potential  future  team  members  who  are  not  currently 
engaged  in  an  appraisal 

When  appraisal  method  training  will  be  delivered  to  more  than  one  appraisal 
team  in  a  single  event,  the  SEI  must  be  notified  in  writing  (e.g.,  via  email),  at 
least  30  days  prior  to  the  first  day  of  training.  When  method  training  is  delivered 
in  this  way,  care  must  be  exercised  to  ensure  that  confidentiality  of  information  is 
maintained  between  organizations. 
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1.3.4  Prepare  Team 


Parameters 
and  Limits 
(continued) 


Method  training  delivered  to  groups  of  potential  future  team  members  must 
cover  the  complete  set  of  tailoring  options  and  allowable  variations  for  the 
method  to  prepare  them  for  a  range  of  situations  they  are  likely  to  encounter  on 
future  appraisals.  When  method  training  is  to  be  delivered  in  this  way,  the  SEI 
must  be  notified,  in  writing  (e.g.,  via  email),  at  least  30  days  prior  to  the  first  day 
of  training. 

Team  members  who  have  previously  received  SCAMPI  A  team  training  are  not 
automatically  qualified  to  participate  on  an  appraisal  without  first  attending 
method  training.  In  such  cases,  the  appraisal  team  leader  is  required  to 
understand  the  nature  of  the  training  delivered  previously  and  the  adequacy  of 
that  training  for  the  appraisal  at  hand. 

There  must  be  at  least  one  event  where  the  team  gathers  as  a  group  for  the 
purpose  of  establishing  team  norms  and  make  operational  decisions  about  how 
the  team  will  work  for  the  appraisal  at  hand.  Even  if  all  team  members  have 
previously  been  trained  in  the  method,  there  must  be  a  team  orientation  session 
that  brings  the  team  together  in  order  to  identify  potential  issues  with  team 
operation. 

Any  training-related  waivers  must  be  documented  in  the  appraisal  plans. 

Individuals  who  are  not  SEI-certified  SCAMPI  Lead  Appraisers  or  People  CMM 
appraisers  may  not  deliver  appraisal  method  training. 

Due  to  the  confidentiality  required  during  an  appraisal  and  the  cohesiveness 
needed  to  participate  in  appraisal  activities,  observers  are  not  permitted  to 
participate  in  the  appraisal  processes.  The  only  exception  is  an  observer  who  is 
certified  by  the  SEI. 
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1.3.4  Prepare  Team 


Implementation 

Guidance 


The  team  training  event  is  a  good  place  to  review  the  appraisal  plan  with 
appraisal  team  members,  having  sent  it  to  them  in  advance  of  their  arrival.  This 
event  provides  the  orientation  for  the  entire  appraisal  that  all  appraisal  team 
members  must  execute  their  roles  appropriately.  This  event  also  is  in  keeping 
with  the  “Provide  appraisal  plan  to  relevant  stakeholders  for  review”  required 
practice  in  activity  1.2.7,  Obtain  Commitment  to  Appraisal  Plan. 


Additionally,  the  team  training  event  is  an  opportunity  to  conduct  activity  1.5.1, 
Perfonn  Readiness  Review.  The  assembled,  trained  appraisal  team  can  then 
appropriately  assess  the  organization’s  readiness  for  the  appraisal  and  validate 
the  reasonableness  of  appraisal  plan. 


Implementation 

Guidance 

Training  in 
the  Reference 
Model 


A  typical  model  training  course  is  delivered  in  three  days.  The  successful 
completion  of  appraisal  reference  model  training  precedes  training  in  the 
appraisal  method.  There  is  no  “aging”  requirement  for  when  this  model  training 
was  received,  but  the  appraisal  team  leader  ensures  that  each  team  member  has 
adequate  reference  model  understanding,  and  takes  remedial  action  if  necessary. 
Attendance  at  model  training  is  recorded  by  the  training  instructor  and  provided 
to  the  SEI,  in  accordance  with  the  terms  of  instructor  authorization.  This  action 
establishes  a  record  in  a  database  which  makes  that  trained  individual  eligible  to 
be  added  to  the  appraisal  team  in  the  SEI  Appraisal  System  (SAS). 


For  appraisals  that  include  higher  levels  (i.e.,  maturity  levels  4  and  5),  team 
members  may  benefit  from  receiving  additional  training  on  this  subject  matter. 
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1.3.4  Prepare  Team 


Implementation 

Guidance 

Training  in 
the  Appraisal 
Method 


Implementation 

Guidance 

Familiarization 
with  the 
Appraisal  Plan 


A  typical  delivery  of  appraisal  team  training  takes  two-and-a-half  to  three  days. 
More  or  less  time  may  be  necessary,  depending  on  the  relative  experience  of  the 
appraisal  team  members. 

Exercises  in  appraisal  techniques  and  team  development  are  used  to  reinforce 
the  skills  that  will  be  important  during  conduct  of  the  appraisal.  It  is 
recommended  that  exercises  be  used  that  are  appropriate  for  the  organizational 
unit  being  appraised.  Where  sufficient  organizational  artifacts  exist,  “live”  data 
can  be  collected  and  used  in  training  exercises  where  appropriate.  Just-in-time 
training  can  also  be  used  to  re-emphasize  method  concepts  at  appropriate  points 
in  the  appraisal  process  during  which  the  skills  will  be  utilized. 

Appraisal  team  training  materials  are  tailored  to  fit  team  needs  and  objectives  of 
the  specific  appraisal.  Tailoring  provides  opportunities  to 

•  provide  insight  into  the  context,  objectives,  and  plans  of  the  particular 
appraisal 

•  communicate  team  members’  assigned  roles  and  responsibilities 

•  identify  tailoring  of  SCAMPI  A  for  the  upcoming  appraisal 

•  acquaint  the  team  with  the  organizational  unit’s  characteristics  and 
documentation 

•  focus  on  skills  that  may  be  more  critical  to  the  upcoming  appraisal,  such  as 
the  ability  to  facilitate  interviews 

It  is  recommended  that  this  training  be  provided  within  60  days  of  the  appraisal. 
The  appraisal  team  leader  typically  provides  method  training  using  materials 
available  in  the  SCAMPI  Lead  Appraiser  kit,  but  other  delivery  options  are  also 
acceptable  (as  described  above).  Although  alternative  training  options  can 
provide  some  advantages  and  efficiencies  for  method  training,  there  are  also 
potential  negative  consequences.  Familiarization  with  the  particular  tailoring 
options  of  a  given  event,  or  unique  aspects  of  the  organization  may  not  be 
adequately  covered  in  a  large  training  session  that  includes  many  teams. 
Regardless  of  how  method  training  is  delivered  to  the  team  members, 
opportunities  for  team  building  are  provided  to  coalesce  the  team  and  bring  the 
team  up  to  speed  on  the  specifics  of  the  appraisal  being  planned. 


Method  training  and  team  building  provide  good  opportunities  to  establish  team 
familiarity  with  the  appraisal  plan.  This  familiarity  includes  such  items  as 
appraisal  objectives,  organizational  scope,  appraisal  reference  model  scope,  and 
the  schedule,  resources,  and  constraints  for  conducting  the  appraisal.  Team 
member  input  can  be  obtained  to  refine  or  complete  the  contents  of  the  appraisal 
plan. 


CMU/SEI-2011-HB-001  |  93 


1.3.4  Prepare  Team 


Implementation 

Guidance 

Analysis  of 

Objective 

Evidence 


Analysis  of  the  objective  evidence  provided  by  the  appraised  organization,  such 
as  questionnaire  responses  or  worksheets  summarizing  objective  evidence,  can  be 
accomplished  following  or  as  an  integrated  part  of  appraisal  team  preparation  and 
training. 

Demonstrations  or  exercises  using  the  data  collection  tools  and  methods  planned 
for  the  appraisal  provide  appraisal  team  members  with  an  opportunity  to  practice 
techniques  for  data  recording,  verification,  and  analysis.  These  tools  and  methods 
may  include  mechanisms  such  as  wall  charts,  spreadsheets,  or  data  reduction 
tools.  The  more  familiarity  and  comfort  obtained  with  these  tools  in  advance,  the 
greater  the  savings  in  team  efficiency  during  the  Conduct  Appraisal  phase. 
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1.3.4  Prepare  Team 


Implementation 

Guidance 

Roles  and 
Responsibilities 


The  appraisal  team  leader  assigns  and  explains  team  member  roles  and 
responsibilities  to  be  performed  during  the  appraisal.  Typical  roles  to  be 
assigned  include 

Appraisal  Coordinator:  The  appraisal  coordinator  handles  logistics  and 
provides  technical,  administrative,  and  logistical  support  to  the  appraisal  team 
leader.  This  support  usually  includes  activities  such  as  coordinating 
schedules,  notifying  participants,  arranging  adequate  facilities  and  resources, 
obtaining  requested  documentation,  and  arranging  catering.  He  or  she  may 
also  coordinate  or  provide  clerical  support  to  the  team.  This  role  is  often 
assigned  to  one  or  more  members  of  the  organizational  unit.  The  appraisal 
coordinator  may  be  one  of  the  appraisal  team  members,  or  this  role  may  be 
assigned  to  other  site  personnel. 

Librarian:  The  librarian  manages  the  inventory  of  appraisal  documents, 
coordinates  requests  for  additional  documentation  evidence,  and  returns 
documents  at  the  end  of  the  appraisal.  This  role  can  be  filled  by  an  appraisal 
team  member  or  by  a  member  of  the  support  staff. 

Process  area  or  basic  unit  mini-teams:  mini-team  members  take  the  lead  for 
data  collection  in  assigned  process  areas  or  basic  units.  They  ensure  that 
information  collected  during  a  data  gathering  session  covers  their  process 
areas  or  basic  units,  request  additional  information  needed  relative  to  their 
process  areas  or  basic  units,  and  record  the  work  performed  by  individual 
appraisal  team  members  pertaining  to  their  process  areas  or  basic  units. 

Mini-teams  typically  consist  of  two  or  three  members.  Mini-team 
assigmnents  can  be  made  based  on  several  factors,  including  related  process 
areas  (e.g.,  process  area  categories)  and  a  mix  of  mini-team  members  (e.g., 
discipline  experience  and  appraisal  experience). 

Facilitator:  The  facilitator  conducts  interviews. 

Timekeeper:  The  timekeeper  is  responsible  for  tracking  time  and  schedule 
constraints  during  interviews  and  other  activities. 

Appendix  C  contains  additional  information  about  the  roles  and 
responsibilities  of  appraisal  team  members. 

♦ 


CMU/SEI-2011-HB-001  |  95 


1.4  Obtain  and  Inventory  Initial  Objective  Evidence 


Purpose 

Obtain  infonnation  that  facilitates  site-specific  preparation  and  an  understanding 
of  the  implementation  of  model  practices  across  the  organizational  unit.  Identify 
potential  issues,  gaps,  or  risks  to  aid  in  refining  the  plan.  Strengthen  the  appraisal 
team  members’  understanding  of  the  organization’s  operations  and  processes. 
Note:  The  data  collection  approach  (discovery,  managed  discovery,  and/or 
verification)  is  a  tailoring  option  of  SCAMPI  A.  If  the  discovery  option  is 
chosen,  there  may  be  limited  objective  evidence  to  inventory  at  this  stage  of  the 
appraisal.  If  the  managed  discovery  option  is  chosen,  an  initial  data  call  results 
in  a  set  of  evidence  that  is  evaluated,  and  followed  by  successive  data  calls  based 
on  remaining  evidence  gaps. 

Entry  Criteria 

•  draft  appraisal  plan 

•  sponsor  authorization  to  proceed 

•  availability  of  practice  implementation  data  for  organizational  unit 

Inputs 

•  practice  implementation  data  for  organizational  unit 

•  identified  participants 

Activities 

1.4.1  Obtain  Initial  Objective  Evidence 

1.4.2  Inventory  Objective  Evidence 

Outputs 

•  data  inventory  results  (e.g.,  data  availability  summaries) 

•  identification  of  additional  information  needed 

•  initial  set  of  objective  evidence 

Outcome 

•  Initial  objective  evidence  has  been  collected,  organized,  and  recorded. 

•  Potentially  important  areas  of  needed  information  have  been  noted. 

•  The  team  has  a  deeper  understanding  of  the  organizational  unit’s  operations 
and  processes. 

•  The  team  is  ready  to  make  detailed  plans  for  data  collection. 

Exit  Criteria 

•  All  objective  evidence  captured  during  this  activity  has  been  recorded  for 
later  use. 

•  High-priority  areas  for  additional  data  collection  have  been  identified. 

•  The  level  of  sufficiency  of  the  inventory  of  objective  evidence  to  support  the 
appraisal  is  determined. 
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1.4  Obtain  and  Inventory  Initial  Objective  Evidence 


Key  Points 

Gather  high-leverage  objective  evidence.  The  amount  of  initial  objective 
evidence  provided  by  the  organization  will  determine  the  proportion  of  evidence 
that  remains  to  be  discovered  (versus  verified)  during  the  appraisal.  Efficient  and 
effective  time  spent  in  collection,  inventory,  and  verification  of  evidence  is  a  key 
performance  objective  for  the  Conduct  Appraisal  phase. 

Tools  and 
Techniques 

•  Automated  support,  including  data  reduction  tools,  may  be  available  to  make 
the  data  inventory  activity  more  efficient. 

•  Breaking  into  mini-teams  to  inventory  data  related  to  specific  process  areas 
is  a  way  to  help  ensure  completeness  of  the  data. 

Metrics 

•  the  number  of  practices  for  which  complete  objective  evidence  is  available 

•  the  calendar  time  and  effort  expended  for  this  activity  compared  to  the 
planned  values 

Verification  and 
Validation 

•  Where  the  team  includes  members  of  the  appraised  organization,  these 
members  should  be  used  to  help  understand  the  initial  objective  evidence 
provided  to  prevent  misinterpretation  of  terms  or  special  conditions. 

•  Inconsistencies  and  contradictions  among  the  items  provided  in  initial 
objective  evidence  should  be  identified  and  recorded  for  resolution. 

Records 

•  Lists  of  information  needed  should  be  maintained  and  used  as  input  to  the 
later  data  collection  activities. 

•  Calendar  time  and  effort  expended  in  this  activity  should  be  recorded  and 
compared  to  the  plan.  These  data  will  be  part  of  the  appraisal  record. 
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1.4  Obtain  and  Inventory  Initial  Objective  Evidence 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


This  process  plays  a  critical  role  in  the  planning  and  preparation  processes.  The 
information  generated  in  this  process  provides  the  most  important  opportunity 
to  reset  expectations  and  plans  with  the  appraisal  sponsor,  if  initial  assumptions 
about  the  availability  of  objective  evidence  turn  out  to  be  in  error.  It  will  also 
provide  the  basis  of  data  collection  planning. 

For  appraisals  that  are  conducted  in  discovery  or  managed  discovery  mode, 
some  activities  in  this  process  may  necessarily  be  combined  with  the  activities 
in  Section  2. 2, Examine  Objective  Evidence. 


The  appraisal  team  leader  works  with  representatives  of  the  organization  to 
obtain  an  initial  data  set  that  provides  input  for  an  inventory  of  the  objective 
evidence  pertaining  to  the  implementation  of  each  practice  among  the  selected 
sample  basic  units  and  support  functions  within  the  appraisal  scope.  This  initial 
data  set  may  be  first  reviewed  by  the  appraisal  team  leader  for  a  high-level 
assessment  of  adequacy  and  completeness.  The  appraisal  team  leader  or 
appraisal  team  then  performs  a  more  detailed  inventory  to  use  as  input  for 
planning  the  data  collection  and  verification  activities  that  will  occur  when  they 
begin  the  Conduct  Appraisal  phase.  Finally,  a  record  is  created  that  reflects  a 
detailed  accounting  of  any  missing  objective  evidence.  This  record  is  used  as 
primary  input  for  the  generation  of  the  data  collection  plan. 


♦ 


CMU/SEI-2011-HB-001  |  98 


1.4.1  Obtain  Initial  Objective  Evidence 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  appraisal  team  leader  will  request  that  the  organization  provide  detailed 
data  on  the  implementation  of  practices  in  the  organization.  The  appraisal  team 
leader  is  free  to  specify  the  format  to  be  used  and  the  level  of  detail  to  be 
provided,  knowing  that  anything  that  is  not  provided  in  advance  must  be 
collected  later  in  the  appraisal  process.  There  are  no  minimum  requirements  set 
by  the  method  with  respect  to  completeness  or  detail  in  this  initial  data  set. 
Before  the  appraisal  outputs  can  be  created,  the  team  must  verify  objective 
evidence  for  each  instantiation  of  each  practice  within  the  scope  of  the 
appraisal.  For  detailed  requirements  on  the  sufficiency  of  data,  refer  to  process 
2.4,  Verify  Objective  Evidence. 

The  data  collection  approach  influences  the  conduct  of  this  activity.  Minimal 
evidence  may  be  available  in  a  discovery-based  appraisal.  A  pre-determined  set 
of  high-leverage  evidence  requested  in  an  initial  evidence  call  is  provided  in  a 
managed  discovery  appraisal.  In  a  verification-based  appraisal,  the 
organization  may  provide  a  completely  populated  database  of  objective 
evidence  mapped  to  model.  The  appraisal  team  leader  must  allow  an 
opportunity  for  the  organization  to  provide  evidence  based  on  the  chosen  data 
collection  approach. 


The  appraisal  team  leader  or  designee  shall 

•  obtain  data  reflecting  the  implementation  of  model  practices  among  sample 
basic  units  and  support  functions  within  the  organizational  unit 


At  a  minimum,  the  organization  must  provide  a  list  of  artifacts  that  are  relevant 
to  understanding  the  processes  in  use  among  sample  basic  units  and  support 
functions  within  the  organizational  unit,  unless  a  discovery-based  appraisal  has 
been  planned.  This  list  must  be  mapped  to  the  model  practices  that  are  in  the 
scope  of  the  appraisal. 

This  activity  does  not  replace  the  activities  in  Section  2.2,  Examine  Objective 
Evidence. 
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1.4.1  Obtain  Initial  Objective  Evidence 


Implementation 

Guidance 


Whether  collected  using  questionnaires,  reviewing  artifacts,  attending 
presentations,  observing  demonstrations,  or  conducting  interviews,  the  data 
used  for  an  appraisal  is  related  to  the  practices  of  the  appraisal  reference  model. 
For  every  practice  within  the  reference  model  scope  of  the  appraisal,  and  for 
every  instance  of  each  practice,  objective  evidence  is  used  as  the  basis  for 
appraisal  team  determinations  of  the  extent  to  which  the  practice  is 
implemented.  Objective  evidence  that  substantiates  practice  implementation 
includes  the  following: 

•  artifacts,  which  represent  a  tangible  form  of  objective  evidence  indicative  of 
work  being  performed  that  represents  either  the  primary  output  of  a  model 
practice  or  a  consequence  of  implementing  a  model  practice.  Sufficient 
artifacts  demonstrating  and  corroborating  that  the  work  is  being  done  are 
necessary  to  verify  the  implementation  of  associated  model  practices. 

•  affirmations,  which  are  oral  statements  that  confirm  whether  or  not  a  model 
practice  has  been  implemented.  Those  who  have  implemented  (or  should 
have  implemented)  a  model  practice  provide  affirmations  to  the  appraisal 
team  during  an  interactive  forum  which  the  appraisal  team  controls. 

Prior  to  the  data  collection  activities  carried  out  by  the  appraisal  team,  an  initial 
data  set  is  usually  created  by  the  appraised  organization.  This  data  set  contains 
descriptions  of  the  objective  evidence  available  for  the  team  to  examine, 
complete  with  references  to  artifacts  and  identification  of  the  personnel  who  can 
provide  relevant  affirmations.  The  data  set  provides  the  baseline  of  objective 
evidence  for  the  appraisal.  Most  organizations  experienced  in  process 
improvement  will  already  have  this  type  of  data  on  hand,  as  they  will  have  used 
it  to  track  their  improvement  progress. 

Artifacts  may  be  obtained  as  hard  copies,  soft  copies,  or  hyperlinks  to  where 
these  documents  reside  in  a  web-based  environment.  If  hyperlinks  are  used,  the 
accessibility  of  artifacts  via  these  links  should  be  verified  in  the  appraisal 
environment.  For  example,  appraisal  team  access  could  be  inhibited  by  invalid 
references  or  firewalls. 

The  initial  data  set  forms  the  basis  for  planning  data  collection  activities, 
including  interviews,  demonstrations,  and  presentations  on  site.  Any  objective 
evidence  that  is  not  identified  in  advance  of  the  team’s  arrival  must  be  sought 
by  the  team  members  once  they  begin  the  Conduct  Appraisal  phase.  This 
process  of  discovering  whether  and  how  the  organization  has  addressed  a  given 
practice  in  the  model  can  be  quite  time  consuming  and  it  is  often  difficult  to 
predict  how  long  it  will  take. 


♦ 
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1.4.2  Inventory  Objective  Evidence 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  inventory  of  the  initial  data  set  provides  critical  new  information  for  the 
overall  planning  of  the  appraisal  and  forms  the  basis  for  the  detailed  data 
collection  plan  that  must  be  developed  before  the  Conduct  Appraisal  phase.  The 
inventory  of  initial  objective  evidence  at  this  stage  is  focused  primarily  on  the 
adequacy  and  completeness  of  infonnation  in  the  context  of  the  data  collection 
approach  chosen  (discovery,  managed  discovery,  and/or  verification)  and  the 
implications  for  future  data  collection.  The  results  of  this  activity  are  the  primary 
basis  for  determining  the  extent  of  additional  evidence  collection  to  be  performed 
in  the  future  appraisal  activities. 


The  appraisal  team  leader  and/or  designees  shall 

•  examine  the  initial  set  of  objective  evidence  provided  by  the  organizational 
unit,  unless  a  discovery-based  appraisal  has  been  selected 

•  determine  the  extent  to  which  additional  objective  evidence  is  needed  for 
adequate  coverage  of  model  practices 


Information  provided  by  the  organizational  unit  must  be  detailed  enough  to 
understand  the  extent  to  which  each  type  of  objective  evidence  (i.e.,  artifacts  and 
affirmations)  is  available  for  each  process  instantiation  for  each  model  practice 
within  the  scope  of  the  appraisal.  This  initial  review  of  objective  evidence 
identifies  model  practices  for  which  the  team  may  decide  it  has 

•  strong  objective  evidence 

•  no  objective  evidence 

•  conflicting  objective  evidence 

•  inconsistent  objective  evidence 

•  insufficient  objective  evidence 

In  the  process  of  inventorying  the  available  objective  evidence,  potential 
alternative  practices  must  be  considered.  Any  objective  evidence  for  such 
practices  must  be  determined  as  early  as  possible.  See  Appendix  B,  Alternative 
Practice  Identification  and  Characterization  Guidance,  for  information  on 
identifying  acceptable  alternative  practices. 

Key  artifacts  are  identified  that  can  be  used  to  gain  insight  regarding  a  number  of 
model  practices.  These  artifacts  contain  potential  high-leverage  information  and 
may  be  good  candidates  for  early  review  by  team  members. 

Identify  additional  objective  evidence  needs  relative  to  model  practices  for  each 
basic  unit  or  support  function  within  the  scope  of  the  appraisal.  This  activity  does 
not  replace  the  activities  in  Section  2.2,  Examine  Objective  Evidence. 
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1.4.2  Inventory  Objective  Evidence 


Implementation 

Guidance 


Members  of  the  team  may  choose  to  summarize  the  extent  of  available 
information  at  the  discretion  of  the  appraisal  team  leader.  However,  the  objective 
of  this  activity  is  to  determine  how  much  additional  data  team  members  must 
gather  to  complete  their  work.  It  is  recommended  that  the  appraisal  team  leader 
establish  an  expectation  with  the  sponsor  that  the  results  of  this  activity  will  form 
the  basis  for  a  revised  schedule  estimate.  If  the  initial  objective  evidence  is 
lacking  in  completeness  and  detail,  the  team  will  need  to  seek  more  information 
during  the  Conduct  Appraisal  phase,  unless  corrective  actions  are  taken  before 
that  time. 

It  is  important  to  keep  all  stakeholders  focused  on  the  fact  that  SCAMPI  A  is 
intended  as  a  benchmarking  appraisal.  This  method  is  not  well  suited  for 
organizations  that  have  a  limited  understanding  of  CMMI.  Such  organizations 
may  not  yet  have  a  clear  idea  of  how  the  practices  described  in  the  reference 
models  ought  to  be  implemented  to  meet  their  specific  business  needs.  Deciding 
on  a  reasonable  implementation  of  the  practices,  and  working  to  ensure  that  they 
are  enacted  throughout  the  organization,  are  activities  that  precede  a 
benchmarking  appraisal.  A  different  type  of  appraisal  (Class  B  or  C)  is  probably 
more  valuable  if  the  objective  of  the  sponsor  is  to  begin  the  process  of 
understanding  what  CMMI  could  mean  for  the  organization.  It  is  not  reasonable 
to  schedule  a  two-week  appraisal  and  expect  to  collect  all  of  the  data  required  for 
benchmarking  during  the  Conduct  Appraisal  phase. 

Whatever  data  collection  approach  is  chosen  (discovery,  managed  discovery, 
and/or  verification),  the  appraisal  team  leader  often  reviews  the  initial  data  set 
provided  by  the  organization  prior  to  assembling  the  team  for  its  first  meeting  to 
identify  areas  where  additional  data  will  be  needed  and  to  assess  the  feasibility  of 
the  planned  appraisal  schedule.  This  readiness  review  should  be  conducted  prior 
to  finalizing  the  appraisal  schedule,  and  may  comprise  a  “go/no-go”  decision  for 
the  appraisal  in  some  situations.  The  appraisal  team  may  then  review  the  initial 
objective  evidence  in  more  detail  (typically  toward  the  end  of  the  team- training 
event)  to  begin  formulating  plans  for  how  missing  evidence  will  be  collected. 
This  preliminary  readiness  review  is  the  basis  for  the  data  collection  plan,  which 
is  described  in  the  next  process,  1 .5,  Prepare  for  Appraisal  Conduct. 
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1.4.2  Inventory  Objective  Evidence 


Implementation 

Guidance 

(continued) 


The  appraisal  team  leader  generates  a  list  of  additional  information  needed.  The 
results  of  the  inventory  of  initial  objective  evidence  are  documented  as  an  input 
to  the  data  collection  plan.  The  use  of  an  integrated  appraisal  tool  to  annotate 
the  set  of  initial  objective  evidence  will  permit  the  automated  tracking  of 
information  needs,  and  will  aid  in  the  compilation  of  a  detailed  data  collection 
plan.  Where  the  completeness  of  initial  objective  evidence  is  insufficient  to 
conduct  the  appraisal  under  the  original  schedule,  the  results  of  this  activity 
form  an  important  basis  for  renegotiating  the  appraisal  schedule  in  some  cases. 


The  adequacy  of  objective  evidence  relative  to  model  practices  is  typically 
determined  using  a  software  tool  of  some  sort,  either  one  built  for  use  on 
appraisals  or  a  spreadsheet  template.  However,  paper  forms  and  wall  charts 
may  be  used  if  preferred. 


♦ 
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1.5  Prepare  for  Appraisal  Conduct 


Purpose 

Ensure  readiness  to  conduct  the  appraisal,  including  confirmation  of  the 
availability  of  objective  evidence,  appraisal  team  commitment,  logistics 
arrangements,  risk  status  and  associated  mitigation  plans.  Plan  and  document  data 
collection  strategies. 

Entry  Criteria 

•  Sponsor  commitment  to  proceed  with  the  appraisal  has  been  documented. 

•  Appraisal  objectives  and  constraints  have  been  documented. 

•  Initial  objective  evidence  has  been  received  and  an  inventory  has  been 
completed. 

•  Appraisal  logistics  and  risks  have  been  documented. 

•  The  appraisal  team  is  trained. 

Inputs 

•  appraisal  plan 

•  initial  data  collection  plan 

•  data  base  of  objective  evidence  mapped  to  model  practices 

•  initial  objective  evidence  inventory 

•  data  collection  status 

Activities 

1.5.1  Perform  Readiness  Review 

1.5.2  Re -Plan  Data  Collection 

Outputs 

•  revised  appraisal  plan 

•  updates  to  the  data  collection  plan  as  required 

Outcome 

•  updated  plans 

•  team  member  awareness  of  data  status  and  needs 

Exit  Criteria 

•  The  team  is  ready  to  conduct  the  appraisal. 

•  Logistical  arrangements  are  confirmed. 

•  All  preparations  for  data  collection  by  the  team  have  been  made  and  the  data 
collection  plan  has  been  revised  to  reflect  current  understanding. 

•  The  data  collection  plan  is  revised  based  on  readiness  review  results  in 
preparation  for  the  conduct  appraisal  phase. 
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1 .5  Prepare  for  Appraisal  Conduct 


Key  Points 

Performance  of  at  least  one  readiness  review  resulting  in  the  appraisal  team 
leader’s  and  sponsor’s  joint  decision  to  continue  the  appraisal  as  planned,  re¬ 
plan  the  appraisal,  or  cancel  the  appraisal  is  paramount  to  the  successful 
conduct  of  the  appraisal.  The  data  collected  are  the  most  important  input  the 
team  receives.  Careful  planning,  disciplined  tracking  against  the  plan,  and 
effective  corrective  actions  are  cornerstones  to  success  in  this  process. 

Tools  and 
Techniques 

The  use  of  a  spreadsheet  to  record  and  track  the  data  collection  plan  is  a 
common  technique.  A  matrix  showing  the  practices  of  the  model,  or  questions 
to  be  asked,  arrayed  on  the  vertical  axis  and  the  sources  of  information  arrayed 
on  the  horizontal  axis  provides  a  simple  planning  and  tracking  tool.  A  number 
of  vendor-provided  data  management  tools  are  available  as  well. 

Metrics 

•  estimated  and  tracked  calendar  time  and  effort  for  this  activity 

•  planned  and  actual  number  of  data  sources  per  practice 

•  planned  and  tracked  number  of  scripted  questions  used  per  interview 

•  planned  and  tracked  number  of  scripted  questions  used  per  process  area 

•  percentage  of  planned  coverage  achieved  per  data  collection  event  or 
process  area 

•  number  of  changes  to  the  appraisal  plan  relative  to  the  appraisal  team, 
logistics,  and  risks 

Verification  and 
Validation 

The  data  collection  plan  should  be  summarized  and  reviewed  with  the  team  to 
ensure  that  appraisal  requirements  will  be  successfully  implemented  if  the  plan 
is  carried  forward.  Experienced  appraisal  team  leaders  will  use  historical  data  to 
assess  the  feasibility  of  (and  risks  associated  with)  the  data  collection  plan. 

Records 

Planned  and  actual  coverage  of  practices  and  process  areas  across  the  set  of  data 
collection  activities  should  be  recorded.  These  data  support  future  estimates  and 
corrective  actions  during  data  collection  activities. 
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1 .5  Prepare  for  Appraisal  Conduct 


interfaces 
with  Other 
Processes 


Summary  of 
Activities 


The  data  collection  plan  is  an  essential  element  of  the  appraisal  plan.  The 
activities  described  here  rely  on  the  results  of  an  inventory  of  the  initial 
objective  evidence  to  derive  a  plan  and  set  of  strategies  for  accomplishing  the 
data  collection  needed  to  meet  the  objectives  of  the  appraisal.  The  data 
collection  plan  developed  through  these  activities  is  reviewed  and  revised  on  a 
continual  basis  throughout  the  appraisal.  Dynamically  managing  the  inventory 
of  data  on  hand,  the  list  of  data  needed,  and  the  available  data  collection 
opportunities  are  processes  critical  to  the  success  of  the  appraisal. 


The  activities  in  this  process  serve  to  (a)  ensure  readiness  to  conduct  the 
appraisal,  (b)  establish  the  initial  planning  baseline  for  the  acquisition  of 
objective  evidence,  and  (c)  update  the  plan  to  account  for  information  acquired 
and  unexpected  developments.  Since  SCAMPI  A  is  a  data-intensive  method, 
the  conduct  of  these  activities  in  accordance  with  the  descriptions  provided  is 
essential  to  the  successful  use  of  the  appraisal  method. 


♦ 
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1.5.1  Perform  Readiness  Review 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  purpose  of  the  readiness  review  is  to  determine  whether  or  not  the  appraisal 
team  and  appraised  organization  are  ready  to  conduct  the  appraisal  as  planned, 
and  in  the  time  allocated.  The  readiness  review  addresses  several  aspects  of 
readiness  to  conduct  the  appraisal:  data  readiness,  team  readiness,  logistics 
readiness,  and  appraisal  risk  status.  The  readiness  review  will  result  in  a  decision 
to  continue  as  planned,  re-plan  or  reschedule,  or  cancel  the  appraisal.  The 
appraisal  team  leader  and  sponsor  are  responsible  for  the  decision  and 
determining  the  conditions  under  which  to  proceed. 


The  appraisal  team  leader  or  designee  shall 

•  determine  whether  the  objective  evidence  for  each  instance  of  each  practice 
in  the  appraisal  scope  is  adequate  to  proceed  with  the  appraisal  as  planned 
(refer  to  activity  1.4.2,  Inventory  Objective  Evidence) 

•  determine  whether  the  appraisal  team  is  prepared  to  conduct  the  appraisal 
(refer  to  activity  1.3.4,  Prepare  Team) 

•  ensure  the  appraisal  logistics  (e.g.  facilities,  equipment,  and  participant 
availability)  have  been  arranged  and  confirmed  (refer  to  activity  1.2.5,  Plan 
and  Manage  Logistics) 

•  review  identified  appraisal  risks  to  determine  status  and  impact  to 
conducting  the  appraisal  as  planned  (refer  to  activity  1.2.6,  Document  and 
Manage  Risks) 

•  review  the  feasibility  of  the  appraisal  plan  in  light  of  data  readiness,  team 
readiness,  logistics  readiness,  and  overall  risk 


The  number  of  readiness  reviews  planned  and  their  dates  must  be  documented  in 
the  data  collection  plan. 

Explicit  criteria  for  detennining  readiness  must  be  established  by  the  appraisal 
team  leader.  These  criteria  must  be  documented  in  the  data  collection  plan.  At 
least  one  readiness  review  must  be  conducted. 

The  conduct  of  a  readiness  review  may  trigger  the  start  of  the  90  day  constraint 
for  accomplishing  phase  II  activities.  If  team  members  perform  document  review 
during  the  readiness  review  (for  the  purpose  of  data  collection  to  support 
characterization)  or  if  any  practice  characterizations  are  determined  —  then  the 
readiness  review  starts  the  “90  day  clock”  for  conducting  phase  II  activities.  If 
the  plan  calls  for  such  document  review  or  characterization,  then  the  entire  team 
must  participate  in  the  readiness  review  where  those  activities  are  carried  out. 

A  readiness  review  may  not  to  be  used  to  identify  weaknesses  in  the 
organization’s  implementation  with  the  intent  to  fix  them  prior  to  the  beginning 
of  the  Conduct  Appraisal  phase. 
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1.5.1  Perform  Readiness  Review 


Parameters 
and  Limits 
(continued) 


If  the  performance  of  readiness  review  for  a  SCAMPI  A  is  integrated  with  the 
performance  of  a  class  B  or  class  C  appraisal,  then  the  following  constraints  apply: 

•  The  phase  II  start  date  of  the  class  B  or  class  C  appraisal  is  taken  as  the  start  of  the 
90  day  period  of  performance  for  phase  II  of  the  SCAMPI  A  appraisal 

•  The  draft  appraisal  plan  for  the  SCAMPI  A  must  be  written  prior  to  the  start  of 
phase  I  of  the  class  B  or  class  C  appraisal. 


The  performance  of  class  B  and  class  C  appraisals  in  advance  of  the  SCAMPI  A  does 
not  require  performance  of  readiness  review  activities.  The  constraints  above  apply 
only  in  situations  where  readiness  review  activities  are  carried  out  along  with  the  class 
B  or  class  C  appraisal  activities. 
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1.5.1  Perform  Readiness  Review 


Implementation 

Guidance 


More  than  one  readiness  review  might  be  needed.  One  should  be  performed  early 
in  the  planning  phase,  long  enough  in  advance  to  give  the  organization  time  to 
collect  any  additional  objective  evidence  and  for  the  appraisal  team  leader  to 
address  any  logistics  problems,  team  issues,  or  other  critical  appraisal  risks  to 
support  a  more  successful  appraisal.  Another  should  be  performed  once  the 
objective  evidence  has  been  gathered  and  the  appraisal  is  ready  to  start.  This 
review  may  be  conducted  in  conjunction  with  the  team  training  event. 

The  appraisal  team  leader  should  lead  the  readiness  review.  Recommended 
participants  include  at  least  one  representative  from  each  appraisal  mini-team,  the 
appraisal  coordinator,  and  any  additional  organizational  unit  representatives 
desired. 

Data  readiness  should  address  what  data  is  available,  what  data  is  still  needed, 
and  how  and  where  additional  data  will  be  obtained. 

Recommended  minimum  criteria  for  data  readiness  include  the  following: 

•  There  are  no  significant  coverage  gaps  in  the  evidence-to-practice  mapping 
provided  by  the  organization. 

•  Artifacts  identified  in  the  objective  evidence  databases  are  accessible. 

•  The  state  of  readiness  and  completeness  is  consistent  with  the  duration  of  the 
planned  Conduct  Appraisal  phase. 

Thresholds  for  the  sufficiency  of  data  should  be  established  as  targets  to  be  met 
at  the  readiness  review.  For  example,  an  80  percent  threshold  may  be  used  to 
initiate  replanning  at  the  final  readiness  review.  That  is,  the  appraisal  team  leader 
establishes  an  expectation  with  the  sponsor  that,  if  more  than  20  percent  of  the 
objective  evidence  is  missing  at  the  time  of  team  training,  the  appraisal  must  be 
re-planned.  Flowever,  the  primary  objective  is  reducing  the  risk  that  there  will  be 
insufficient  objective  evidence  to  make  the  determinations  required  by  the 
appraisal  plan  in  the  time  allotted. 

Objective  evidence  for  all  basic  units  and  support  functions  sampled  to  represent 
the  organizational  unit  should  be  reviewed  to  assess  appropriateness  and 
applicability  to  the  practice. 

Objective  evidence  for  alternative  practices  should  be  considered.  See  Appendix 
B,  Alternative  Practice  Identification  and  Characterization  Guidance,  for 
information  on  identifying  acceptable  alternative  practices. 

Team  readiness  should  address  whether  the  appraisal  team  is  prepared  to  conduct 
the  appraisal.  The  appraisal  team  leader  should  determine  whether  the  team 
members  are  adequately  trained  and  the  mini-teams  are  operating  effectively  and 
efficiently.  If  necessary,  the  appraisal  team  leader  may  change  mini-team 
membership,  add  resources,  or  change  assigmnents  to  optimize  team 
performance. 
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1.5.1  Perform  Readiness  Review 


Implementation 

Guidance 

(continued) 


Logistics  readiness  should  address  whether  the  necessary  logistics  arrangements 
have  been  made  for  the  Conduct  Appraisal  phase.  The  appraisal  team  leader 
should  review  the  logistics  arrangements  to  determine  whether  appropriate 
facilities  have  been  reserved,  the  necessary  equipment  will  be  available,  and  the 
appraisal  participants  have  been  contacted  and  will  be  available  during  the 
Conduct  Appraisal  phase. 

A  summary  of  the  inventory  of  objective  evidence  and  readiness  to  proceed 
should  be  reviewed  with  the  sponsor  or  his/her  designee.  If  insufficient 
objective  evidence  is  available  or  if  any  other  aspect  of  appraisal  readiness  is 
not  met,  the  appraisal  team  leader  may  need  to  initiate  replanning  in  light  of 
newly  discovered  constraints  (e.g.,  insufficient  data  to  support  the  appraisal  as 
planned).  Refer  to  activity  1.1.3,  Determine  Appraisal  Constraints.  The  criteria 
for  adequacy  will  depend  on  where  the  readiness  review  occurs  in  the  schedule, 
and  the  chosen  data  collection  approach  (discovery,  managed  discovery,  and/or 
verification)  that  is  being  sought  for  the  Conduct  Appraisal  phase  of  the 
appraisal. 

The  readiness  review  is  a  key  event  whose  impact  should  not  be 
underestimated.  Failure  to  ensure  that  all  aspects  of  appraisal  readiness  (i.e., 
data,  team,  logistics,  and  overall  risk)  are  reviewed  to  determine  the  impact  on 
the  appraisal  plan  can  have  grave  consequences  for  the  appraisal  team  during 
the  Conduct  Appraisal  phase.  The  Conduct  Appraisal  phase  may  include  long 
hours,  exhaustion,  extensive  ad  hoc  data  collection  (i.e.,  discovery),  or  the 
inability  to  achieve  appraisal  objectives  within  defined  estimates  and 
constraints. 


♦ 
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1.5.2  Re-Plan  Data  Collection 


Activity 

Description 

The  data  collection  plan  is  updated  as  required  during  the  conduct  of  the 
readiness  review  or  during  the  appraisal  itself  as  objective  evidence  is  found,  or 
as  new  sources  of  information  are  uncovered.  The  activity  described  in  this 
section  refers  to  a  more  substantial  change  in  the  plan,  which  is  expected  to  be  a 
rare  occurrence  in  practice.  If  during  the  conduct  of  an  appraisal  the  team 
discovers  that  their  assumptions  about  the  availability  of  objective  evidence  are 
substantially  incorrect,  the  appraisal  team  leader  may  renegotiate  the  appraisal 
plan  with  the  sponsor. 

Required 

Practices 

The  appraisal  team  leader  or  designee  shall 

•  review  the  current  inventory  of  objective  evidence  and  detennine  model 
practices  for  which  the  objective  evidence  is  inadequate  relative  to  the 
appraisal  plan 

•  revise  the  data  collection  plan  as  necessary  based  on  the  appraisal  status  and 
availability  of  objective  evidence 

•  renegotiate  the  appraisal  plan  with  the  sponsor  if  the  appraisal  cannot 
proceed  as  planned 

Parameters 

and  Limits 

None 
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1.5.2  Re-Plan  Data  Collection 


Implementation 

Guidance 


This  activity  is  not  a  substitute  for  tactical  decisions  about  where  and  how  to 
find  objective  evidence.  The  intent  of  this  activity  is  to  respond  to  a  major  gap 
between  expected  data  and  actual  data. 

Major  gaps  between  expected  and  actual  data  may  occur,  for  example,  as  a 
result  of  the  following: 

•  inaccurate  assumptions  about  the  availability  of  objective  evidence 

•  content  of  artifacts  or  information  from  interviews  not  providing  significant 
amounts  of  the  information  required  and  other  sources  not  being  planned 

•  unexpected  absence  of  multiple  key  interviewees 

•  unanticipated  delays  in  the  implementation  of  new  processes 

•  major  customer-driven  emergencies  for  one  or  more  of  the  sampled  basic 
units  or  support  functions 

This  activity  serves  as  a  “pressure  valve”  of  sorts  for  the  appraisal.  The  pressure 
to  perform  the  appraisal  under  unrealistic  conditions  can  lead  to  a  severe 
degradation  in  the  quality  of  the  appraisal  outputs.  Carefully  planning  for 
contingencies  and  communicating  them  to  the  sponsor  help  to  protect  the 
standards  that  must  be  met  in  the  performance  of  an  appraisal.  Clearly 
documenting  the  data  collection  plan,  and  regularly  monitoring  the  availability 
of  data  compared  to  that  plan,  support  effective  risk  mitigation. 

When  this  activity  must  be  employed  to  recover  from  an  unrealistic  expectation, 
the  documentation  reflecting  the  assumptions  made  during  planning,  as  well  as 
concrete  facts  about  what  is  or  is  not  available,  are  used  to  renegotiate  with  the 
appraisal  sponsor.  This  need  to  renegotiate  is  one  of  the  reasons  why  a  detailed 
appraisal  plan,  with  the  sponsor’s  signature,  is  a  required  artifact  for  the 
conduct  of  a  SCAMPI  A  appraisal. 


♦ 
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2  Conduct  Appraisal 

2.1  Prepare  Participants 


Purpose 

Ensure  that  appraisal  participants  are  appropriately  informed  of  the  appraisal 
process,  purpose,  and  objectives  and  are  available  to  participate  in  the 
appraisal  process. 

Entry  Criteria 

•  Data  collection  has  been  planned. 

•  The  sponsor  has  approved  the  appraisal  plan. 

•  The  appraisal  team  is  trained  and  is  familiar  with  the  appraisal  plan. 

Inputs 

•  appraisal  plan 

Activities 

2.1.1  Conduct  Participant  Briefing 

Outputs 

*  prepared  appraisal  participants 

Outcome 

At  the  end  of  this  process,  appraisal  participants  are  prepared  to  provide 
relevant  information  to  the  appraisal  team  and  have  confirmed  their 
participation. 

Exit  Criteria 

A  participant  briefing  has  been  conducted  and  appraisal  participants  are 
prepared  to  participate. 

Key  Points 

Inform  members  of  the  organization  who  participate  in  the  appraisal  of  their 
roles,  and  expectations  of  the  sponsor  and  appraisal  team. 

Tools  and 
Techniques 

•  presentation  tools 

•  video  teleconferencing  facilities 

Metrics 

•  planned  and  actual  number  of  participants  briefed 

Verification  and  .  feedback  from  appraisal  participants  on  their  readiness  to  perform  their 


Validation 

role(s) 

Records 

•  participants  briefed  compared  to  the  plan 
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2.1  Prepare  Participants 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


This  process  includes  informing  appraisal  participants  of  the  appraisal  process, 
its  purpose  and  objectives,  and  their  roles  in  the  appraisal.  At  a  minimum,  this 
communication  is  addressed  in  a  single  opening  briefing  at  the  start  of  the 
Conduct  Appraisal  phase.  Alternatively,  it  might  be  addressed  initially  during 
the  Plan  and  Prepare  phase,  but  with  a  short,  additional  opening  briefing  to  all 
participants — again  at  the  start  of  the  Conduct  Appraisal  phase — to  introduce 
the  team  and  reiterate  the  appraisal  goals.  In  any  event,  it  must  occur  prior  to 
the  conduct  of  any  interviews. 


The  activities  in  this  process  serve  to  prepare  participants  for  the  appraisal. 
Participants  are  prepared  prior  to  their  participation  to  ensure  they  are  aware  of 
their  roles  in  the  appraisal,  confirm  their  availability,  and  prepare  for  their 
participation. 


♦ 
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2.1.1  Conduct  Participant  Briefing 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Implementation 

Guidance 


Members  of  the  organization  who  participate  in  the  appraisal  must  be  informed 
of  their  role  and  the  expectations  the  sponsor  and  appraisal  team  have.  This 
communication  is  typically  accomplished  through  a  briefing  in  which  the 
appraisal  team  leader  provides  an  overview  of  the  appraisal  process,  purpose, 
and  objectives.  Specific  information  about  the  scheduled  events  and  the 
locations  where  they  occur  is  also  communicated  during  this  presentation,  as 
well  as  through  ongoing  contact  between  the  appraisal  coordinator  and  the 
members  of  the  organization. 


The  appraisal  team  leader  and/or  designees  shall 

•  brief  appraisal  participants  on  the  appraisal  process 

•  provide  orientation  to  appraisal  participants  on  their  roles  in  the  appraisal 


Participants  must  reconfirm  their  availability  to  participate  in  the  appraisal. 


The  preparation  of  appraisal  participants  may  be  accomplished  using  a  video 
conference  or  teleconference  if  desired. 

Depending  on  the  appraisal  usage  mode  (e.g.,  supplier  selection  versus  internal 
process  improvement),  various  types  of  communication  may  be  used.  In  the 
internal  process  improvement  usage  mode,  the  importance  of  management 
sponsorship  within  the  organization  will  likely  lead  the  appraisal  team  leader  to 
work  with  senior  management  to  help  demonstrate  commitment  to  the  appraisal 
process  as  well  as  the  process  improvement  work  that  will  follow.  In  the 
supplier  selection  usage  mode,  the  possibility  of  the  same  team  visiting  multiple 
organizations  adds  coordination  tasks  and  communication  channels  as  well. 


♦ 
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2.2  Examine  Objective  Evidence 


Purpose 

Examine  information  about  the  practices  implemented  in  the  organization  and 
relate  the  resultant  data  to  the  appraisal  reference  model.  Perform  the  activity  in 
accordance  with  the  data  collection  plan.  Take  corrective  actions  and  revise  the 
data  collection  plan  as  needed. 

Entry  Criteria 

•  Data  collection  has  been  planned,  and  the  plan  documented. 

•  The  sponsor  has  approved  the  appraisal  plan. 

•  The  appraisal  team  is  trained  and  is  familiar  with  the  appraisal  plan. 

•  Participants  have  been  informed  about  the  appraisal  process  and  their  roles  in 
it. 

Inputs 

•  Appraisal  data 

initial  objective  evidence 

documented  practice  implementation  gaps,  if  any 

documented  exemplary  practice  implementation,  if  any 

feedback  on  preliminary  findings  (if  that  point  in  the  timeline  has  been 

reached) 

•  Data  collection  plan 

appraisal  schedule 
affirmation  schedule 

artifact  list 

new  interview  questions 

Activities 

2.2.1  Examine  Objective  Evidence  from  Artifacts 

2.2.2  Examine  Objective  Evidence  from  Affirmations 

Outputs 

•  updated  appraisal  data 

•  updated  data  collection  plan 

Outcome 

After  the  final  iteration  of  this  process,  the  team  has  sufficient  data  to  create  or 
revise  preliminary  findings  and  to  make  judgments  about  the  implementation  of 
practices,  as  well  as  the  satisfaction  of  specific  and  generic  goals. 

Exit  Criteria 

The  coverage  of  the  appraisal  reference  model  and  the  organizational  scope  has 
been  achieved,  and  the  team  is  ready  to  produce  the  appraisal  outputs. 
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2.2  Examine  Objective  Evidence 


Key  Points 

The  appraisal  team  reviews  the  objective  evidence  that  has  been  gathered 
according  to  the  data  collection  plan.  They  evaluate  this  evidence  against  the 
reference  model  to  determine  if  it  is  appropriate  and  adequate  to  support  the 
implementation  of  model  practices.  The  data  collection  plan  is  modified  if 
additional  information  is  needed  to  complete  this  task.  Effective  contingency 
planning  and  the  use  of  work  aids  to  monitor  progress  are  key  points  to  consider. 
The  team  must  be  able  to  focus  on  examining  the  most  relevant  information 
available,  rather  than  be  distracted  by  a  mission  to  find  new  evidence. 

Tools  and 
Techniques 

Wall  charts  and  other  visual  aids  are  often  used  to  display  the  results  of  data 
collection  activities.  Electronic  tools  are  prevalent  among  experienced  appraisal 
team  leaders  and  can  be  effective  for  continually  monitoring  and  updating  the 
inventory  of  objective  evidence. 

Metrics 

Tracking  the  actual  coverage  obtained,  as  compared  to  the  planned  coverage,  in 
each  data  collection  activity  facilitates  timely  corrective  actions  where  they  are 
needed.  The  most  critical  resource  during  an  appraisal  is  time.  Using  a 
timekeeper  during  data  collection  and  verification  activities  provides  feedback  on 
team  performance.  Recording  the  actual  duration  of  planned  events  helps  the 
team  in  taking  actions  to  recover  from  unexpected  events. 

Verification  and  The  appraisal  method  provides  detailed  verification  and  validation  procedures  for 


Validation 

objective  evidence.  They  are  described  in  process  2.4,  Verify  Objective 

Evidence,  and  2.5,  Validate  Preliminary  Findings. 

Records 

Work  aids  used  to  record  and  track  the  progress  of  data  collection  activities  are 
retained  for  traceability  and  provide  an  important  input  to  a  final  report 
describing  the  appraisal,  if  the  sponsor  has  requested  a  final  report.  The  duration 
and  effort  required  for  specific  data  collection  events  can  be  recorded  to  provide 
useful  historical  data  for  planning  subsequent  appraisals. 
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2.2  Examine  Objective  Evidence 


interfaces 
with  Other 
Processes 


Summary  of 
Activities 


The  activities  that  provide  the  team  with  the  data  needed  to  produce  reliable 
appraisal  outputs  are  perhaps  the  most  visible  part  of  the  appraisal  process  from 
the  perspective  of  the  appraised  organization.  For  this  reason,  SCAMPI  A 
places  a  heavy  emphasis  on  methodically  planning  and  tracking  the  data 
collected  during  an  appraisal.  The  initial  objective  evidence  collected  early  in 
the  process  allows  team  members  to  analyze  the  state  of  information  available 
at  the  earliest  stages  of  the  appraisal  and  narrow  their  search  for  new 
information.  This  early  work  serves  to  facilitate  an  efficient  use  of  time.  An 
explicit  understanding  of  what  information  is  needed  and  how  that  information 
will  be  used  therefore  drives  the  activities  associated  with  this  process. 


The  members  of  the  team  continually  manage  the  data  collected  previously  and 
target  new  data  collection  activities  to  fill  known  information  needs. 

Instruments  can  be  used  early  in  the  appraisal  process  and  often  provide  leads  to 
be  pursued  through  other  data  collection  activities.  Presentations  are  sometimes 
used  to  provide  a  flexible  interactive  forum  where  members  of  the  organization 
can  explain  important  infonnation  about  the  practices  implemented  in  the 
organization.  Artifacts  provide  the  most  explicit  and  lasting  representation  of 
practice  implementation  in  the  organization,  and  the  team  uses  them  to 
understand  how  practices  in  the  reference  model  are  implemented.  Finally, 
interviews  are  used  as  the  most  dynamic  data  collection  technique,  allowing  for 
branching  among  related  topics  and  the  explanation  of  contextual  information 
that  affects  the  implementation  of  practices  as  well  as  alternative  practices. 

The  appraisal  activities  conducted  for  each  of  these  data  collection  sources  are 
similar: 

•  Determine  if  the  information  obtained  is  acceptable  as  objective  evidence. 

•  Relate  the  objective  evidence  to  corresponding  practices  in  the  appraisal 
reference  model. 

•  Relate  the  objective  evidence  to  the  appropriate  part  of  the  appraised 
organizational  unit  (i.e.,  the  sample  basic  unit  or  support  function). 

•  Determine  the  extent  to  which  model  practices  have  been  implemented 
within  the  organizational  unit. 

♦ 
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2.2.1  Examine  Objective  Evidence  from  Artifacts 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


A  substantial  portion  of  the  data  used  by  appraisal  team  members  is  derived  from 
artifacts  (see  MDD  glossary)  that  demonstrate  or  support  the  implementation  of 
model  practices.  Artifact  review  is  an  effective  means  to  gain  detailed  insight 
about  the  implementation  of  practices  in  the  organizational  unit.  However, 
without  a  clear  focus  on  the  data  being  sought,  artifact  review  can  consume  a 
great  deal  of  time  as  team  members  sometimes  attempt  to  read  everything  in 
hopes  that  something  useful  will  be  discovered.  Likewise,  if  artifact  review  is  not 
well  planned,  the  team  will  be  more  likely  to  accept  artifact(s)  that  do  not 
completely  address  or  support  the  implementation  of  the  model  practices  in 
scope.  Objective  evidence  obtained  from  artifacts  and  from  other  sources  is 
documented  in  process  2.3,  Document  Objective  Evidence,  and  verified  in 
process  2.4,  Verify  Objective  Evidence. 


The  appraisal  team  shall 

•  update  the  inventory  of  artifacts  used  as  a  source  of  objective  evidence 

•  review  information  obtained  from  artifacts  and  determine  if  it  is  acceptable 
as  objective  evidence 

•  determine  the  model  practices  corresponding  to  the  objective  evidence 
obtained  from  artifacts 

•  determine  the  portions  of  the  organizational  unit  that  correspond  to  the 
objective  evidence  obtained  from  artifacts 

•  review  artifacts  and  determine  the  extent  to  which  model  practices  have  been 
implemented  in  the  organizational  unit 


The  appraisal  team  must  evaluate  the  content  of  artifacts  to  determine  how  they 
support  model  practice  implementation. 

If  the  appraisal  team  performs  any  part  of  this  activity  during  any  readiness 
review  or  other  appraisal  preparation  activity,  the  Conduct  Appraisal  phase  has 
begun,  and  the  appraisal  must  be  completed  within  90  days. 

Artifacts  used  as  objective  evidence  must  have  been  created  or  revised  prior  to 
the  start  of  the  Conduct  Appraisal  phase.  The  team  may  request  to  view  artifacts 
that  result  from  recurring  activities,  even  if  the  activity  occurred  after  the  start  of 
the  appraisal.  In  no  case  would  an  artifact  created  after  the  start  of  the  appraisal 
be  accepted  as  the  only  artifact  demonstrating  the  implementation  of  a  practice. 
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2.2.1  Examine  Objective  Evidence  from  Artifacts 


Implementation 

Guidance 


The  inventory  should  be  sufficient  to  summarize  the  objective  evidence  used  as  a 
basis  for  appraisal  ratings  generated,  as  required  by  the  appraisal  record  described 
in  activity  3.2.2,  Generate  Appraisal  Record.  Much  of  the  inventory  contents  can 
be  obtained  from  the  mapping  data  or  instruments  obtained  from  the 
organizational  unit,  such  as  the  objective  evidence  database  or  questionnaires. 

The  inventory  can  be  used  to  maintain  a  list  of  artifacts  reviewed  or  additional 
artifacts  requested  from  the  organizational  unit.  This  inventory  is  created  in 
activity  1.4.2,  Inventory  Objective  Evidence,  and  is  a  key  work  product  of  data 
collection  planning. 

One  or  more  team  members  will  seek  data  for  every  practice  in  the  appraisal 
reference  model  scope  of  the  appraisal  through  artifact  review.  This  review  does 
not  require  an  artifact  for  every  practice,  as  any  given  artifact  or  set  of  artifacts  is 
likely  to  provide  data  relevant  to  multiple  practices.  Reference  Section  2.4.1, 
Verify  Objective  Evidence,  for  objective  evidence  sufficiency  rules.  In  a 
verification-based  appraisal,  the  location  of  objective  evidence  relating  to  every 
practice  should  be  recorded  in  advance  of  the  team’s  arrival  at  the  site  where  the 
appraisal  will  occur.  Organizations  with  established  improvement  infrastructures 
typically  maintain  this  type  of  information  to  track  their  improvement  efforts 
against  the  model. 

In  a  discovery-based  appraisal,  the  team  will  need  to  discover  the  links  between 
the  reference  model  and  the  organization’s  implemented  practices,  and  will 
therefore  require  more  time  to  perform  the  appraisal.  In  a  managed  discovery 
appraisal,  the  appraisal  team  calls  for  a  predetermined  set  of  high-leverage 
artifacts,  and  uses  a  succession  of  focused,  iterative  calls  for  evidence  to  fill 
remaining  gaps.  Refer  to  Appendix  E,  Managed  Discovery  for  more  information 
on  managed  discovery  and  comparison  to  discovery  and  verification  data 
collection  approaches. 

Artifacts  indicative  of  work  being  performed  can  take  many  forms  including 
documents,  presentations,  and  demonstrations.  A  single  artifact  can  reflect  the 
implementation  of  one  or  more  model  practices. 

Documents  are  a  collection  of  data  regardless  of  the  medium  on  which  it  is 
recorded  (hard  copy,  soft  copy,  or  accessible  via  hyperlinks  in  a  web-based 
environment,  such  as  wikis  or  blogs).  Documents  can  be  work  products  reflecting 
the  implementation  of  one  or  more  model  practices.  These  documents  typically 
include  work  products  such  as  organizational  policy,  product  components, 
process  descriptions,  plans,  risk  registers  and  specifications. 
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2.2.1  Examine  Objective  Evidence  from  Artifacts 


Implementation 

Guidance 

(continued) 


A  demonstration  or  walkthrough  of  the  tool  by  the  person  who  uses  the  tool  as 
part  of  performing  their  process  is  an  acceptable  artifact.  The  demonstration 
includes  review  of  the  content  of  the  tool  for  the  respective  basic  units  that  use 
the  tool  relative  to  model  practice  implementation  (e.g.  a  demonstration  of 
fields  in  a  requirements  management  tool  showing  traceability  from  a 
requirement  to  its  derived  requirement). 


Note:  This  approach  has  the  added  benefit  of  being  able  to  be  used  as  an 
affirmation  collection  session  (see  activity  2.2.2,  Examine  Objective  Evidence 
from  Affirmations). 


Members  of  the  appraisal  team  observing  a  presentation  that  is  developed  by  a 
basic  unit  team  member  as  a  consequence  of  their  work  in  implementing  a 
practice  is  another  acceptable  artifact  (e.g.,  attending  the  presentation  of  a  basic 
unit  review  by  a  project  manager).  The  basic  unit  review  package,  together  with 
its  presentation  is  an  example  of  an  artifact. 

Presentations  created  for  an  appraisal  team  as  an  aid  in  understanding  the 
organizational  processes  and  implementation  of  model  practices  is  not  an  output 
of  implementing  a  process  and  therefore  is  not  a  valid  artifact  demonstrating 
implementation  of  a  model  practice.  However,  such  a  presentation,  delivered  to 
the  appraisal  team  in  an  environment  that  allows  for  interaction  and  follow-up, 
can  serve  as  affirmations  of  practice  implementation  (see  activity  2.2.2, 

Examine  Objective  Evidence  from  Affirmations). 


It  is  recommended  that  the  appraisal  team  evaluate  objective  evidence  threads 
or  sequences  to  better  understand  the  extent  of  practice  implementation  across 
multiple  model  practices  or  process  areas.  An  example  (applicable  to  CMMI 
models)  of  evaluating  objective  evidence  threads  across  process  areas  is 
establishing  and  monitoring  work  group  schedules.  The  schedule  is  established 
in  a  project  plan  is  typically  part  of  an  integrated  master  schedule  (Project 
Planning).  Periodic  progress  reviews  of  work  group  tasks  are  held  to  monitor 
performance  against  the  established  schedule  (Project  Monitoring  and  Control). 
Deviations  are  recorded  in  a  report  and  actions  are  taken  to  align  performance 
with  work  group  needs  and  requirements  (Integrated  Project  Management).  See 
activity  1.1.4,  Determine  Appraisal  Scope,  for  rules  on  sampling. 


♦ 
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2.2.2  Examine  Objective  Evidence  from  Affirmations 


Activity 

Description 


Required 

Practices 


Affirmations  are  used  to  confirm  or  support  implementation  of  model  practices 
within  the  organizational  scope  of  the  appraisal.  Interviews,  as  well  as  other 
affirmation  techniques  (e.g.,  presentations,  questionnaires,  and  demonstrations) 
are  held  with  managers  and  practitioners  responsible  for  the  work  being 
performed.  The  appraisal  team  uses  affirmations  to  understand  how  the  processes 
are  implemented  and  to  probe  areas  where  coverage  of  model  practices  is  needed. 

Affirmations  are  a  required  and  necessary  component  of  a  SCAMPI  A  appraisal 
in  all  usage  modes.  The  criteria  for  the  amount  of  affirmation  objective  evidence 
to  be  collected  are  described  in  activity  1.1.4,  Determine  Appraisal  Scope.  These 
criteria  drive  the  development  of  the  initial  affirmation  strategy  documented  in 
the  data  collection  plan  described  in  activity  1.2.3,  Develop  Data  Collection  Plan. 
The  appraisal  team  leader  works  with  the  team  to  schedule  the  most  appropriate 
affirmation  techniques  for  the  situation. 

As  objective  evidence  is  gathered  throughout  the  appraisal,  the  data  collection 
plan  is  revised  as  necessary.  By  using  focused  investigation  techniques,  the  need 
for  affirmations  may  be  either  increased  or  diminished,  as  long  as  the  criteria  for 
affirmations  are  satisfied. 

Objective  evidence  obtained  from  affirmations  and  other  sources  is  documented 
in  process  2.3,  Document  Objective  Evidence,  and  verified  in  process  2.4,  Verify 
Objective  Evidence. 


The  appraisal  team  shall 

•  establish  and  maintain  an  inventory  of  affirmations  used  as  a  source  of 
objective  evidence 

•  conduct  affirmation  activities  to  obtain  information  that  may  be  used  as 
objective  evidence 

•  review  information  obtained  from  affirmations  and  determine  if  it  is 
acceptable  as  objective  evidence 

•  determine  the  model  practices  corresponding  to  the  objective  evidence 
obtained  from  affirmations 

•  determine  the  portions  of  the  organizational  unit  that  correspond  to  the 
objective  evidence  obtained  from  affirmations 

•  review  information  from  affirmations  and  determine  the  extent  to  which 
model  practices  have  been  implemented  in  the  organizational  unit 
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2.2.2  Examine  Objective  Evidence  from  Affirmations 


Parameters 
and  Limits 


All  SCAMPI  A  appraisals  must  use  affirmations  as  a  source  of  information  on  the 
extent  to  which  practices  have  been  implemented  in  the  organizational  unit  and 
within  the  sampled  basic  units  and  support  functions. 

Sessions  where  affirmations  are  sought  and  collected  must  include  at  least  two 
members  of  the  appraisal  team  designated  by  the  appraisal  team  leader. 

Whenever  virtual  methods  such  as  video  conferences,  teleconferences,  and  other 
similar  technologies  are  used  for  affirmations,  the  appraisal  team  leader  must 
ensure  that  these  methods  do  not  compromise  the  integrity  or  accuracy  of  the 
appraisal  activities  or  the  appraisal  results. 

Sufficient  affirmations  must  be  obtained  to  meet  the  coverage  specified  in  the  data 
collection  plan. 

Steps  must  be  taken  to  ensure  open  communication  during  affirmations  by 
addressing  potential  issues  among  interviewees  and  team  members  (e.g.,  presence 
of  supervisors  or  process  owners). 

The  rules  of  confidentiality  and  the  expected  use  of  appraisal  data  must  be 
communicated  to  every  interviewee. 

If  the  appraisal  team  performs  any  part  of  this  activity  during  any  readiness 
review  or  other  appraisal  preparation  activity,  the  Conduct  Appraisal  phase  has 
begun,  and  the  appraisal  must  be  completed  within  90  days. 
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2.2.2  Examine  Objective  Evidence  from  Affirmations 


Implementation 

Guidance 


Affirmations  are  typically  sought  from  logical  groupings  within  an  organization 
to  provide  insight  into  the  depth  of  the  implementation  by  specific  instance  of  the 
model  practices  being  examined.  Groupings  include  basic  units  (e.g.  projects, 
teams,  or  workgroups),  function  within  the  groups  (e.g.  mangers  or  service 
providers),  or  support  functions  (e.g.  quality  assurance,  configuration 
management,  human  resources,  IT,  training). 

A  variety  of  affirmation  collection  techniques  are  employed  during  an  appraisal 
including  interviews,  presentations,  demonstrations,  and  questionnaires. 
Interviews  are  used  as  the  most  dynamic  data  collection  technique,  allowing  for 
exploration  among  related  topics. 

Presentations  can  be  used  as  a  flexible  interactive  forum  where  members  of  the 
organization  performing  the  work  can  explain  important  information  about  the 
practices  implemented  in  the  organization.  . 

An  interactive  demonstration  or  walkthrough  of  the  tool  by  the  person  who  uses 
the  tool  as  part  of  performing  their  process  is  another  affirmation  technique. 

Use  interactive  virtual  methods  (e.g.,  video  conference,  teleconference,  e-mail 
exchange,  instant  messaging,  or  other  similar  technology)  to  conduct  affirmations 
at  a  distance. 

Interviews  provide  the  most  flexible  source  of  detailed  data.  Direct  interaction 
with  people  who  enact  the  practices  being  investigated  allows  the  team  to  seek 
detailed  information  and  to  understand  the  interrelationships  among  various 
practices.  Detailed  information  to  address  specific  data  collection  needs  can  be 
sought  and  verified  in  real  time. 

Management  personnel  are  typically  interviewed  individually,  or  grouped 
according  to  basic  unit  or  support  function.  The  focus  of  the  discussion  in  these 
interviews  will  therefore  be  scoped  to  a  particular  basic  unit  or  support  function, 
rather  than  across  the  sampled  basic  unit  or  support  functions. 

Functional  area  representatives  (FARs)  are  typically  interviewed  in  a  group 
across  the  basic  units  and  within  the  organizational  scope  of  the  appraisal.  The 
focus  of  the  discussion  in  these  interviews  will  therefore  be  scoped  to  a  particular 
set  of  practices  used  across  the  instantiations  within  the  organizational  scope. 

Request  that  interviewees  bring  a  document  or  other  artifact  with  them  to  their 
interviews  for  a  “show-and-tell”  style  interview.  The  interviewee  explains  how 
the  artifact  is  developed  and  used  to  support  the  work  being  performed. 
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2.2.2  Examine  Objective  Evidence  from  Affirmations 


Implementation 

Guidance 

(continued) 


It  is  important  to  avoid  sampling  interviewees  for  a  session  such  that  two  people 
in  the  same  reporting  chain  (e.g.,  a  superior  and  one  of  his/her  direct  reports)  are 
in  the  same  interview  session.  This  restriction  applies  to  members  of  the 
appraisal  team  as  well.  People  who  have  this  type  of  relationship  with  one 
another  may  be  uncomfortable  with  the  expectation  for  them  to  be  completely 
candid  during  the  interview. 


Samples  of  interviewees  are  typically  grouped  into  categories  that  roughly 
correspond  to  lifecycle  phases,  work  groups,  engineering  disciplines, 
organizational  groupings,  and/or  process  area  affinities.  Interviews  may  include 
individuals  performing  work  related  to  a  single  process  area  or  group  of  process 
areas.  As  stated  previously,  interviews  of  management  personnel  are  typically 
grouped  by  basic  unit  or  support  function,  while  FARs  sampled  for  a  given 
interview  come  from  across  the  organizational  unit. 


Virtual  methods  can  be  used,  but  they  should  be  set  up  to  allow  for  adequate 
interaction  between  the  appraisal  team  members  and  the  appraisal  participants 
and  should  provide  mechanisms  for  the  appraisal  team  to  control  the  interviews 
(e.g.,  provide  the  ability  to  interrupt,  ask  questions,  or  redirect  the  discussion  to 
other  subjects). 
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2.2.2  Examine  Objective  Evidence  from  Affirmations 


There  are  three  basic  forms  of  interviews  used  in  SCAMPI  A.  They  are  described 
below. 

The  most  structured  approach  is  the  standard  interview,  which  is  scheduled  in 
advance  and  employs  a  series  of  scripted  questions.  Each  standard  interview 
typically  involves  interviewees  with  similar  responsibilities  in  the  organization 
(e.g.,  quality  assurance  personnel,  systems  engineers,  help  desk  technicians,  or 
middle  managers).  The  schedule  and  location  of  each  interview  session  is 
communicated  to  the  interviewees  well  in  advance.  Questions  intended  to  elicit 
data  about  particular  practices  are  prepared  and  reviewed  in  advance,  and  the 
team  follows  a  defined  process  for  conducting  the  session. 

Depending  on  the  interview,  the  entire  team  or  specific  mini  teams  may  be 
present  for  these  interviews.  Responsibility  for  tracking  the  coverage  of 
individual  process  areas  is  typically  assigned  to  team  members.  A  single 
questioner  may  lead  the  interview,  with  the  rest  of  the  team  or  portion  of  the 
team  listening  and  taking  notes,  or  the  responsibility  for  asking  questions  may  be 
distributed  among  the  team  members.  In  any  case,  it  is  expected  that  all 
participating  team  members  who  are  not  asking  questions  listen  and  take  notes 
for  all  questions. 

A  set  of  planned  interviews  will  be  defined  during  appraisal  planning.  As  the 
appraisal  progresses  and  the  objective  evidence  accumulates,  the  team  may  find 
it  convenient  to  cancel  one  or  more  of  these  interviews  to  use  the  time  for  other 
activities.  Such  changes  in  the  data  collection  plan  are  made  in  a  way  that  does 
not  violate  the  coverage  criteria  described  in  process  2.4,  Verify  Objective 
Evidence. 


Implementation 

Guidance 

Standard 

Interviews 


Implementation 

Guidance 

On-Call 

Interviews 


A  more  flexible  approach  to  scheduling  interviews  is  available  in  the  form  of  on- 
call  interviews,  a  variant  of  the  standard  interview.  Prospective  interviewees  are 
identified  and  notified  in  advance,  just  as  described  above.  However,  the 
interviews  are  only  held  if  team  members  decide  that  there  is  a  need  and  that  the 
time  will  be  well  spent.  The  prospective  interviewees  are  therefore  asked  to  block 
a  period  of  time  for  such  a  contingency.  These  interviews  need  not  include  the 
entire  appraisal  team  (as  long  as  two  appraisal  team  members  attend),  thus 
permitting  parallel  sessions  with  different  interviewees. 


CMU/SEI-201 1-HB-001  |  126 


2.2.2  Examine  Objective  Evidence  from  Affirmations 


Implementation 

Guidance 

Office  Hours 
Interviews 


Finally,  office  hours  interviews  represent  an  agreement  for  availability  that 
permits  pairs  of  team  members  (per  the  Parameters  and  Limits  of  this  activity), 
to  visit  interviewees  at  their  desks,  cubicles,  or  offices.  As  with  the  on-call 
interviews,  the  prospective  interviewees  block  a  specific  time  period  to  be 
available  on  a  contingency  basis.  Most  prospective  interviewees  will  be  able  to 
continue  with  their  daily  work  and  accommodate  an  interruption  if  the  team 
needs  to  speak  with  them.  Flere  again,  only  if  specific  data  needs  are  identified 
will  the  interview  occur.  The  interviewees  should  be  informed  that  they  may 
receive  only  limited  advanced  notice  for  these  interviews,  although  confirming 
the  interview  at  least  a  day  in  advance  is  a  courtesy  that  should  be  offered 
whenever  possible. 


♦ 
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2.3  Document  Objective  Evidence 


Purpose 

Create  lasting  records  of  the  information  gathered  by  identifying  and  then 
consolidating  notes,  transforming  the  data  into  records  that  document  gaps  in 
practice  implementation  or  exemplary  practice  implementation. 

Entry  Criteria 

Planning  activities  for  the  appraisal  are  complete,  including  the  selection  and 
preparation  of  the  appraisal  team.  At  least  one  data  collection  activity  has  been 
conducted,  and  appraisal-relevant  data  are  available  to  record. 

Inputs 

•  appraisal  data 

•  notes  taken  during  data  collection  activities  (if  applicable) 

•  annotated  worksheets  or  other  work  aids  containing  data  (if  applicable) 

•  gaps  in  practice  implementation  or  exemplary  practice  implementation 
documented  from  previous  activities 

•  data  collection  plan 

Activities 

2.3.1  Take/Review/Tag  Notes 

2.3.2  Record  Presence/ Absence  of  Objective  Evidence 

2.3.3  Document  Practice  Implementation 

2.3.4  Review  and  Update  the  Data  Collection  Plan 

Outputs 

•  updated  appraisal  data 

tagged  notes 

noted  practice  implementation  gaps  (if  any) 
noted  exemplary  implementation  (if  any) 
revised  data  collection  plan  (if  applicable) 
annotated  worksheets 

•  requests  for  additional  data  (artifacts  or  affirmations) 

Outcome 

Individual  team  members  understand  the  data  collected  thus  far,  and  have 
information  to  guide  any  needed  subsequent  data  collection. 

Exit  Criteria 

All  data  from  the  most  recent  data  collection  session  has  been  captured  as  a  new 
baseline  of  practice  implementation  evidence.  The  data-gathering  plans  have 
been  updated  to  reflect  additional  information  needed  and  topics  that  can  be 
removed  from  active  investigation. 
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2.3  Document  Objective  Evidence 


Key  Points 


Tools  and 
Techniques 


Metrics 


Verification  and 
Validation 


Records 


This  process  has  traditionally  been  the  most  difficult  one  to  manage  during  an 
appraisal.  Members  of  the  team  will  tend  to  vary  a  great  deal  in  their  productivity 
and  style  of  work.  The  appraisal  team  leader  must  be  attentive  to  the  progress  of 
each  team  member  and  take  effective  corrective  actions  to  ensure  team  progress. 


Because  of  the  challenging  nature  of  this  activity,  appraisal  team  leaders  tend  to 

have  strong  preferences  for  using  tools  and  techniques  they  have  found  to  be 

successful.  Only  a  high-level  list  of  tools  and  techniques  is  provided  here: 

•  Work  aids  like  wall  charts,  spreadsheet  programs,  and  automated  database 
tools  are  frequently  used  to  help  track  the  status  of  data  collection. 

•  Using  mini-teams,  where  pairs  (or  triplets)  of  team  members  are  assigned 
specific  process  areas  or  specific  basic  units,  is  a  common  practice. 

•  Time  management  is  a  critical  skill  for  this  activity.  Explicitly  reviewing  the 
effort  spent,  in  real  time,  is  a  useful  way  to  focus  the  team. 

•  A  variety  of  techniques  for  structuring  team  notebooks  and  formats  for 
recording  notes  has  been  used. 

•  Team  nonns  regarding  techniques  for  managing  debates  and  divergent  views 
are  important,  and  should  be  made  explicit  well  in  advance. 


As  mentioned  above,  tracking  the  effort  expended  during  this  activity  (in  real 
time)  is  a  valuable  technique  to  manage  the  team’s  time.  The  ability  to  quickly 
leam  the  rate  at  which  each  team  member  works  is  a  skill  that  experienced 
appraisal  team  leaders  develop  using  effort  and  duration  metrics. 


The  method  rules  for  recording  traceability  and  validating  data  provide 
verification  and  validation  of  the  appraisal  data.  Monitoring  progress  and  the 
consensus  decision-making  process,  conducted  by  the  appraisal  team  leader,  also 
serves  as  important  verification  and  validation  activity. 


All  appraisal  data  are  recorded  with  full  traceability  to  information  sources  as 
well  as  the  model  components  to  which  they  pertain.  The  full  detail  in  this 
traceability  contains  sensitive  information  that  should  not  be  provided  to  people 
outside  of  the  appraisal  team.  The  attribution  of  data  to  individuals  must  never  be 
divulged  even  when  detailed  data  are  provided  to  the  organization  for  use  in 
process  improvement. 
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2.3  Document  Objective  Evidence 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


The  mechanics  associated  with  the  recording  and  transcription  of  objective 
evidence  are  described  in  this  section.  There  are  many  links  between  these 
mechanics  and  the  data  collection  process,  as  well  as  the  data  verification  and 
validation  process.  The  data-recording  process  must  support  these  other 
processes,  and  the  tools  used  during  an  appraisal  must  accommodate  these 
linkages.  Typically,  a  database  tool  is  used  to  manage  all  appraisal  data  that 
results  from  the  analysis  of  notes  taken  during  data  collection. 


The  most  basic  representation  of  appraisal  data  is  found  in  the  notes  taken  by 
individual  team  members.  These  notes  are  reviewed  and  are  typically  “tagged” 
or  otherwise  processed  before  their  content  is  transfonned  into  other  lasting 
representations.  The  presence,  absence,  and/or  appropriateness  of  objective 
evidence  is  then  judged  and  recorded  based  on  the  data  collected.  The  scheme 
by  which  this  set  of  records  is  produced  is  an  important  implementation  choice 
made  by  the  appraisal  team  leader,  and  must  be  well  understood  by  the  team. 
Gaps  in  the  implemented  practices  are  also  recorded  in  a  consistent  manner  that 
ensures  traceability.  Finally,  the  data  collection  plan  is  reviewed  in  light  of  the 
changes  in  the  set  of  data  available  to  the  team  and  the  remaining  data  needed  to 
support  reliable  rating  judgments. 


♦ 
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2.3.1  Take/Review/Tag  Notes 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


As  team  members  examine  data  sources,  they  document  what  the  objective 
evidence  is  (e.g.,  referencing  documents,  presentations,  instruments,  and 
interviewee  comments),  as  well  as  why  or  how  the  objective  evidence  meets  the 
intent  of  the  model  practice. 

There  may  be  special  cases  where  team  members  elect  to  record  data  directly  in 
the  objective  evidence  tracking  tool.  In  such  cases,  team  members  may  choose 
not  to  take  notes  (on  paper  or  in  their  notebooks)  that  describe  the  objective 
evidence. 

For  all  interviews  and  presentations,  the  team  members  must  take  notes  that 
capture  the  objective  evidence  before  they  move  to  the  annotation  of  the 
objective  evidence  tracking  tool. 


The  appraisal  team  shall 

•  record  notes  obtained  from  objective  evidence  data-gathering  sessions 

•  relate  notes  to  corresponding  practices  in  the  appraisal  reference  model 


Every  team  member  present  must  take  notes  during  interviews  and  presentations. 
These  notes  must  cover  all  areas  investigated  during  the  interview,  and  are  not 
limited  to  the  process  areas  assigned  to  the  individual  team  member  (i.e., 
everybody  takes  notes  on  everything). 

During  document  reviews,  notes  must  be  taken  to  preserve  specific  context  or 
focused  references,  if  the  rationale  for  accepting  the  objective  evidence  is  not 
self-evident. 
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2.3.1  Take/Review/Tag  Notes 


Implementation 

Guidance 


Implementation 

Guidance 

Taking  Notes 


Implementation 

Guidance 

Reviewing 

Notes 


Implementation 

Guidance 

Tagging  Notes 


The  raw  notes  taken  during  an  appraisal  are  treated  as  confidential  information 
and  may  not  be  provided  to  any  person  outside  of  the  appraisal  team.  Team 
members  are  typically  required  to  destroy  their  notes  in  a  secure  manner  at  the 
conclusion  of  the  appraisal.  This  requirement  ensures  that  the  attribution  of 
detailed  information  to  individuals  in  the  organization  cannot  lead  to 
inappropriate  consequences  following  the  appraisal. 


Team  members  actively  take  notes  during  all  data-gathering  sessions.  The 
purpose  is  to  record,  verbatim,  what  the  information  source  reveals  about  the 
implementation  of  practices  in  the  basic  unit  or  organization. 

Whenever  notes  are  taken  in  a  data-gathering  session,  individual  team  members 
should  review  their  notes  immediately  after  the  conclusion  of  the  session.  The 
review  will  focus  on  tagging  significant  items  that  relate  to  one  or  more  model 
practice(s).  This  review  and  tagging  process  should  occur  within  24  hours  of 
the  data-gathering  session. 


The  context  in  which  the  data  are  provided — be  it  during  an  interview, 
presentation,  or  in  a  document — affects  the  proper  interpretation  of  the  facts. 
For  example,  notes  taken  during  an  interview  are  based  on  a  give  and  take 
between  the  interviewer  and  the  interviewee.  The  threads  of  discussion  often 
provide  a  context  that  may  not  be  reflected  in  a  single  excerpt  from  the  middle 
of  the  interview.  Note-takers  should  review  their  work  to  ensure  that  such 
contextual  information  can  be  preserved  at  least  in  their  recollection,  and 
preferably  through  the  annotation  of  the  notes. 


As  notes  are  reviewed,  team  members  often  use  highlighter  pens  or  annotation 
schemes  to  identify  the  most  salient  excerpts.  The  process  area  and/or  practice 
to  which  the  information  applies  may  be  written  in  colored  ink  over  the  raw 
notes.  All  notes  should  identify  the  data-gathering  session,  and  the  pages  should 
be  numbered  to  preserve  the  sequence  of  information.  For  notes  taken  during 
interviews,  it  is  often  useful  to  draw  a  seating  chart  to  show  where  each  person 
was  sitting  during  the  interview.  Scripts  prepared  in  advance  of  scheduled 
interviews  may  already  be  tagged,  and  can  help  relate  responses  to  appropriate 
sections  of  the  appraisal  reference  model.  Some  interviewee  responses  may  deal 
with  model  practices  other  than  those  targeted  by  a  given  question,  which 
would  still  necessitate  some  additional  tagging. 


♦ 
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2.3.2  Record  Presence/Absence  of  Objective  Evidence 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  presence  or  absence  of  appropriate  objective  evidence  for  each  model 
practice  in  the  scope  of  the  appraisal  is  determined  based  on  information 
obtained  from  data-gathering  sessions.  Annotations  are  recorded  indicating  the 
source,  relevance,  and  coverage  of  objective  evidence  collected.  In  situations 
where  just  referencing  the  data  source  would  not  make  it  obvious  why  the 
objective  evidence  is  appropriate,  a  comment  can  be  added  to  the  annotation.  For 
example,  when  an  alternative  to  the  typical  work  breakdown  structure  is  used,  it 
may  be  necessary  to  document  why  that  alternative  meets  the  intent  of  the  model 
practice.  Adding  comments  to  the  annotations  can  help  to  avoid  rehashing  the 
rationale  for  accepting  the  objective  evidence  multiple  times  during  team 
discussions. 


The  appraisal  team  shall 

•  for  each  instantiation,  record  the  presence  or  absence  of  objective  evidence 
collected  for  each  reference  model  practice  within  the  appraisal  scope 


The  inventory  of  objective  evidence  (be  it  in  electronic  or  paper  form)  is  updated 
to  reflect  what  the  data  imply  about  the  implementation  of  particular  practices. 
For  every  practice  within  the  reference  model  scope  of  the  appraisal,  annotations 
indicating  the  presence  or  absence  of  objective  evidence  will  be  made  throughout 
the  appraisal  conduct.  The  annotation  scheme  used  must  ensure  that  the  record 
reveals  the  following  information: 

•  the  basic  unit  or  support  function  to  which  the  data  apply 

•  the  specific  or  generic  practice  to  which  the  data  apply 

•  the  type  of  objective  evidence  being  recorded  (i.e.,  artifact  or  affirmation) 

•  whether  the  data  imply  the  presence  or  absence  of  the  objective  evidence 

•  whether  the  data  suggest  that  the  objective  evidence  is  appropriate 

•  comments  about  the  appropriateness  of  the  evidence  (if  needed) 

•  whether  or  not  additional  information  is  needed  before  the  team  can 
characterize  the  extent  to  which  the  practice  is  implemented 

•  a  description  of  what  the  evidence  is,  if  such  a  description  was  not  provided 
by  the  organization  in  advance 
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2.3.2  Record  Presence/Absence  of  Objective  Evidence 


Implementation 

Guidance 


This  activity  represents  the  mechanical  aspects  of  processing  appraisal  data,  and 
is  strongly  tied  to  the  activities  described  in  process  2.4,  Verify  Objective 
Evidence.  The  emphasis  of  this  activity  description  is  on  the  steps  needed  to 
update  the  inventory  of  objective  evidence  and  maintain  traceability  to  data 
sources.  The  emphasis  of  the  activity  description  in  Verify  Objective  Evidence 
is  on  the  interpretation  of  data  collected  and  the  sufficiency  of  objective 
evidence  relative  to  the  appraisal  reference  model. 

Team  members  typically  record  the  presence  or  absence  of  appropriate 
objective  evidence  into  tools  such  as  tracking  tables  or  data  consolidation 
worksheets.  Prior  to  the  assignment  of  goal  ratings,  the  entire  team  reviews  the 
status  of  the  objective  evidence  as  reflected  in  the  annotations  made  by  each 
team  member. 

The  data  gathered  during  every  data  collection  session  should  be  related  to  the 
practices  in  use  in  a  basic  unit  (e.g.,  project  or  work  group)  or  support  function 
within  the  organizational  unit.  In  recording  the  presence  or  absence  of  objective 
evidence,  the  intent  is  to  quickly  inventory  the  composite  of  factual 
information.  Elaboration  about  what  the  data  mean  or  how  they  relate  to  other 
important  issues  is  captured  either  in  notes  or  in  the  descriptions  of  practice 
implementation  gaps  crafted  by  team  members. 


♦ 
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2.3.3  Document  Practice  Implementation 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  primary  intent  of  this  activity  is  to  derive,  from  the  objective  evidence 
gathered,  records  that  describe  the  gap  between  what  the  objective  evidence 
shows  and  what  the  team  was  looking  for  to  support  a  claim  that  the  model 
practice  was  implemented.  The  statements  explain  why  the  practice  is  not 
considered  to  be  Fully  Implemented. 

Statements  expressing  exemplary  implementation  of  practices  (i.e.  strengths) 
may  be  recorded  when  objective  evidence  indicates  that  the  implementation  of 
the  practice  is  exceptional.  If  the  organization  is  compliant  with  the  model 
practices,  but  the  implementation  is  not  considered  to  be  exemplary,  no 
statements  are  recorded. 


The  appraisal  team  shall 

•  document  gaps  in  the  basic  unit  or  support  function’s  implemented 
processes  relative  to  appraisal  reference  model  practices 

•  document  exemplary  implementation  in  the  basic  unit  or  support 
function’s  implemented  processes  relative  to  appraisal  reference  model 
practices 


For  any  practice  that  is  characterized  as  something  other  than  Fully 
Implemented,  there  must  be  a  statement  explaining  the  gap  between  what  the 
organization  does  and  what  the  model  expects. 

Statements  of  practice  implementation  gaps,  presented  to  the  organizational 
unit  in  the  form  of  preliminary  findings  for  validation,  must  be  free  of 
references  to  specific  individuals  or  groups,  unless  basic  unit  level  findings  are 
planned  for  the  appraisal. 

Regardless  of  the  medium  used,  statements  describing  practice  implementation 
gaps  or  exemplary  implementation  of  model  practices  must  be  annotated  with 
the  following  identifying  information: 

•  the  model  component  to  which  the  statement  relates  (i.e.,  process  area, 
goal,  and  practice) 

•  the  data  collection  session(s)  in  which  the  information  was  uncovered 

•  the  basic  unit  or  support  function  to  which  the  statement  applies 
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2.3.3  Document  Practice  Implementation 


Implementation 

Guidance 


The  database  used  to  record  the  inventory  of  objective  evidence  may 
incorporate  functionality  to  record  practice  implementation  gaps  and  exemplary 
practice  implementation,  or  a  separate  location  or  tool  may  be  used  if  desired. 
Such  statements  should  be  recorded  at  the  level  of  a  particular  instance  of  a 
model  practice.  These  precursors  to  preliminary  findings  (i.e.,  strengths  and 
weaknesses)  are  more  detailed  and  pointed,  while  all  infonnation  presented 
outside  of  the  team  will  be  aggregated  to  the  goal  and  organizational  unit  level 
of  abstraction.  It  is  not  necessary  to  generate  statements  expressing  adequate 
implementation  of  a  model  practice. 


♦ 
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2.3.4  Review  and  Update  the  Data  Collection  Plan 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Implementation 

Guidance 


This  activity  is  used  to  continuously  monitor  the  state  of  available  objective 
evidence  and  to  select  the  next  tactic  in  the  pursuit  of  obtaining  full  coverage  of 
the  reference  model  scope  and  organizational  scope  of  the  appraisal. 


The  appraisal  team  shall 

•  review  the  inventory  of  objective  evidence  collected  and  the  data  collection 
plan  to  determine  what  additional  objective  evidence  is  still  needed  for 
sufficient  coverage  of  the  appraisal  reference  model  scope 

•  revise  the  data  collection  plan  to  obtain  additional  objective  evidence  for 
instances  where  insufficient  data  are  available  to  judge  the  implementation 
of  appraisal  reference  model  practices 

•  identify  priorities  for  the  upcoming  data  collection  events  and  reevaluate  the 
feasibility  of  the  schedule  in  light  of  the  current  state  of  the  objective 
evidence 


This  activity  must  be  enacted  at  least  once  a  day,  and  a  consolidated  summary  of 
the  appraisal  data  collection  status  must  be  available  to  the  team  at  the  start  of 
each  day  during  which  data  collection  events  are  planned. 


The  data  collection  status  summarizes  the  differences  between  the  objective 
evidence  on  hand  and  the  evidence  needed  to  support  the  creation  of  appraisal 
outputs  (e.g.,  findings  and  ratings).  Annotations  regarding  the  presence  (and 
appropriateness)  of  objective  evidence  allow  the  team  to  inventory  the  state  of 
the  “knowledge  base.”  This  status  then  drives  requirements  for  the  collection  of 
more  data.  Annotation  of  the  inventory  of  objective  evidence  is  described  in 
process  2.4,  Verify  Objective  Evidence. 

The  plan  for  future  data  collection  should  be  revisited  and  updated  as  necessary. 
There  may  be  several  situations  in  which  additional  data  are  required  for  the  team 
to  sufficiently  characterize  the  implementation  of  appraisal  reference  model 
practices.  For  example: 

•  The  process  of  reconciling  new  data  with  the  old  may  identify  conflicts  or 
ambiguities  in  the  data  that  require  clarification. 

•  The  search  for  objective  evidence  may  lead  to  the  discovery  of  one  or  more 
previously  undocumented  practice(s)  in  the  organization. 

•  Attempts  to  confirm  the  use  of  a  particular  practice  or  tool  by  a  basic  unit  or 
support  function  may  have  been  unsuccessful. 
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2.3.4  Review  and  Update  the  Data  Collection  Plan 


Implementation 

Guidance 

(continued) 


Prioritizing  data  needs  and  allocating  data  collection  effort  to  particular  data 
collection  events  are  ongoing  activities  that  the  appraisal  team  leader  is 
responsible  for  overseeing.  The  data  collection  status  summary  may  be 
maintained  by  the  appraisal  team  leader  and  reported  to  the  team  members,  or 
the  appraisal  team  leader  may  elect  to  have  each  mini-team  perform  this  activity 
for  the  process  areas  it  is  assigned. 


Specific  infonnation  needed  to  resolve  ambiguities  or  conflicts  in  the  existing 
data  should  be  documented  for  follow-up  by  one  or  more  members  of  the  team. 
For  detailed  data  items  that  have  a  limited  scope  of  impact,  the  notes  of 
individual  team  members  may  be  adequate  to  document  the  data  needed.  For 
example,  whether  or  not  a  particular  person  is  involved  in  a  meeting,  or  reviews 
a  given  document,  can  be  confirmed  by  a  simple  question  asked  during  an  on- 
call  interview.  Therefore,  a  note  made  by  an  individual  team  member  to  make 
sure  the  question  is  asked  may  suffice. 


In  contrast,  if  conflicting  information  is  uncovered  about  whether  or  not  a  given 
event  occurred  (e.g.,  a  meeting)  more  visibility  of  this  conflict  may  be  needed 
among  the  team  members  to  understand  why  the  information  collected  thus  far 
is  not  internally  consistent.  In  such  a  case,  the  person(s)  responsible  for  the 
process  area  in  which  that  practice  resides  may  need  to  alert  the  team  to  the 
conflicting  data  and  facilitate  a  team  discussion  to  seek  clarity,  as  well  as 
additional  data.  This  potential  conflict  may  lead  to  the  crafting  of  a  specific 
interview  question,  which  is  used  in  a  standard  interview. 


The  data  collection  plan  and  inventory  of  objective  evidence  provide  a  means 
for  the  appraisal  team  to  continuously  monitor  progress  toward  sufficient 
coverage  of  appraisal  reference  model  practices  in  preparation  for  rating. 
Estimates  of  the  additional  data  collection  effort  should  be  regularly  reviewed. 
If  the  feasibility  of  the  appraisal  schedule  is  called  into  question,  a  replanning 
effort  may  be  necessary  (as  described  in  activity  1.5.2,  Re-Plan  Data 
Collection). 


♦ 
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2.4  Verify  Objective  Evidence 


Purpose 

Verify  the  sufficiency  of  objective  evidence  to  determine  the  implementation  of 
model  practices  for  each  instantiation.  Describe  any  strengths  and  weaknesses  in 
the  implementation  of  model  practices.  Each  implementation  of  each  practice  is 
verified  so  that  it  may  be  compared  to  the  practices  of  the  reference  model.  Then 
the  team  characterizes  the  extent  to  which  the  practices  in  the  model  are 
implemented. 

Entry  Criteria 

Objective  evidence  has  been  collected  about  the  implementation  of  practices  in 
the  organization.  Gaps  in  the  implementation  of  model  practices  or  exemplary 
practice  implementation  have  been  identified,  and  the  team  is  ready  to 
characterize  the  extent  to  which  model  practices  (or  acceptable  alternatives  to 
those  practices)  have  been  implemented. 

Inputs 

•  appraisal  plan,  including  schedule  and  participants  for  verification  activities 

•  practice  implementation  gaps  (if  any) 

•  exemplary  practice  implementation  (if  any) 

•  data  collection  plan  specifying  any  additional  information  needed 

Activities 

2.4.1  Verify  Objective  Evidence 

2.4.2  Characterize  Implementation  of  Model  Practices  and  Generate  Preliminary 
Findings 

Outputs 

•  updated  appraisal  data 

strength  statements  (if  any) 
weakness  statements  (if  any) 
annotated  worksheets 

•  updated  appraisal  artifacts 

preliminary  findings 
revised  data  collection  plan 
requests  for  additional  data 

•  practice  characterizations 

instantiation  level 
organizational  unit  level 

Outcome 

The  team’s  confidence  in  the  material  that  will  form  the  basis  for  appraisal 
outputs  is  increased.  Any  critical  deficiencies  in  the  data  on  hand  have  been 
identified  and  actions  to  resolve  these  issues  have  been  initiated. 

Exit  Criteria 

The  team  has  recorded  data  on  the  implementation  of  practices  in  the 
organization,  and  characterized  the  extent  to  which  practices  in  the  model  are 
implemented. 
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2.4  Verify  Objective  Evidence 


Key  Points 

The  data  used  to  formulate  appraisal  outputs  must  be  verified  to  ensure  that  the 
results  of  aggregating  individual  detailed  data  items  will  lead  to  appropriate 
appraisal  outputs. 

Tools  and 
Techniques 

Facilitation  techniques  to  guide  the  team  through  difficult  decisions  are  important 
during  this  activity  (as  they  are  during  the  rating  activity  as  well).  Techniques  to 
enhance  the  credibility  of  the  preliminary  findings  are  also  important. 

Metrics 

Planned  versus  actual  effort  expended  for  this  process  (as  with  all  activities)  will 
assist  in  monitoring  progress  as  well  as  planning  subsequent  appraisals. 

Verification  and  The  appraisal  team  leader  must  ensure  active  participation  in  verification 


Validation 

activities  as  a  way  of  confirming  that  the  verification  process  is  working  as 
intended.  Reactions  to  the  validation  activity  (activity  2.5.1,  Validate  Preliminary 
Findings)  will  provide  feedback  to  help  validate  that  this  activity  was  successful. 

Records 

Characterizations  of  practice  implementation  and  strength/weakness  statements 
will  be  recorded  for  subsequent  use  by  the  team. 
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2.4  Verify  Objective  Evidence 


Interfaces  with 

Other 

Processes 


Summary  of 
Activities 


During  the  conduct  of  an  appraisal,  the  team  gathers  and  analyzes  a  great  deal 
of  detailed  information.  Processes  described  earlier  in  this  document  clarify 
how  data  are  gathered  and  examined.  The  process  described  here  is  focused  on 
understanding  the  information  revealed  by  the  data.  The  processes  described 
after  this  one  are  focused  on  making  reliable  and  valid  rating  judgments  based 
on  the  verified  data. 


The  initial  objective  evidence  provided  by  the  organization  is  used  to 
understand  how  practices  are  intended  to  be  implemented.  Members  of  the 
appraisal  team  then  seek  information  to  confirm  that  the  intended  practices  are 
indeed  implemented.  This  first  verification  activity  (2.4.1,  Verify  Objective 
Evidence)  may  reveal  gaps  or  strengths  in  the  actual  implementation  that  are 
not  apparent  in  the  initial  objective  evidence  provided  by  the  organization. 

The  next  verification  activity  (2.4.2,  Characterize  Implementation  of  Model 
Practices  and  Generate  Preliminary  Findings)  then  compares  the  implemented 
practices  to  the  practices  in  the  reference  model.  This  activity  may  also  reveal 
gaps  in  the  implementation(s)  that  will  later  bear  on  the  ratings  assigned  by  the 
team.  Standard  characterizations  to  capture  the  extent  of  practice 
implementation,  first  at  the  instantiation  level  and  then  at  the  organizational  unit 
level,  are  recorded  by  the  team  with  descriptions  of  gaps  in  implementation. 


♦ 
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2.4.1  Verify  Objective  Evidence 


Activity 

Description 


Required 

Practices 


The  appraisal  team  establishes  a  clear  understanding  of  the  practices 
implemented  in  the  organization  through  the  data  collection  activities  carried  out 
during  the  appraisal.  Typically,  the  organization  provides  a  set  of  objective 
evidence  at  the  beginning  of  the  appraisal  process,  and  the  team  follows  the  data 
collection  plan  to  gather  the  complete  set  of  data  required  to  support  the 
verification  process  described  here.  Parameters  for  adequacy,  coverage  and 
sufficiency  are  specified  here.  Consult  Appendix  F,  Scoping  and  Sampling  in 
SCAMPI  A  Appraisals,  for  more  detailed  illustrations  of  the  flow  of  activities 
relating  to  sampling  and  data  adequacy. 


The  appraisal  team  shall 

•  verify  the  appropriateness  of  artifacts  provided  by  basic  units  or  support 
functions  to  enable  adequate  understanding  of  the  extent  of  implementation 
of  each  practice  within  the  appraisal  reference  model  scope 

•  verify  the  appropriateness  of  affirmations  provided  by  people  from  basic 
units  or  support  functions  to  enable  adequate  understanding  of  the  extent  of 
implementation  of  each  practice  within  the  appraisal  reference  model  scope 

•  verify  that  the  artifacts  and  affirmations  provided  are  sufficient  to  cover  the 
organizational  and  model  scope  of  the  appraisal 
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2.4.1  Verify  Objective  Evidence 


Parameters 
and  Limits 


Implementation 

Guidance 


Data  Adequacy  Rules 

Artifacts:  For  an  artifact  to  be  accepted  as  evidence  of  practice  implementation, 
it  must  be  a  product  or  by-product  of  the  practice  being  examined  by  the 
appraisal  team. 

Affirmations:  For  an  affirmation  to  be  accepted  as  evidence  of  practice 
implementation,  it  must  be  supplied  by  an  individual  who  participated  in  the 
implementation  of  the  practice  being  examined  by  the  appraisal  team. 

Data  Sufficiency:  Verify  that  all  data  (i.e.,  artifacts  and/or  affirmations)  are 
provided  for  all  sampled  basic  units  and  support  functions  for  the  model  scope 
of  the  appraisal,  in  accordance  with  the  coverage  rules  specified  in  Section 
1.1.4,  Determine  Appraisal  Scope,  and  documented  in  the  data  collection  plan. 

Coverage  of  the  Basic  Unit  or  Support  Function:  For  basic  units  or  support 
functions  that  encompass  multiple  disciplines  or  facets  of  work,  ensure  that  the 
objective  evidence  covers  all  aspects  of  the  work  (e.g.,  in  a  systems  and 
software  project,  looking  only  at  evidence  relating  to  software  engineering  is 
inadequate). 


See  Appendix  B,  Alternative  Practice  Identification  and  Characterization 
Guidance,  for  information  on  cases  involving  alternative  practices. 

The  example  work  products  listed  in  the  reference  models  provide  examples  of 
artifacts  that  can  be  used  to  indicate  practice  implementation.  These  are 
examples  only  and  are  not  required;  alternatives  more  appropriate  to  the 
organization  and  the  basic  units  will  typically  be  appropriate. 

Typically,  many  of  the  artifacts  required  to  support  this  verification  are 
provided  in  advance  of  the  Conduct  Appraisal  phase.  The  primary  focus  of  data 
collection  is  to  permit  the  team  to  verify  that  the  intended  practices  are 
implemented  across  the  organizational  unit.  Where  the  implemented  practices 
differ  from  the  intended  practices,  the  objective  evidence  provided  at  the  start  of 
the  appraisal  process  is  annotated  to  more  accurately  reflect  the  implemented 
process  in  the  organization.  These  annotations  are  typically  statements 
describing  a  gap  in  the  implementation  of  a  practice,  some  of  which  will 
eventually  become  weaknesses. 

Only  after  team  members  have  a  clear  understanding  of  the  implemented 
practices  can  they  compare  them  to  the  model  to  characterize  the  extent  to 
which  the  organizational  unit  implements  the  practices  in  the  model  or 
acceptable  alternatives. 


♦ 
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2.4.2  Characterize  Implementation  of  Model  Practices  and  Generate 
Preliminary  Findings 

Activity  Once  objective  evidence  on  practice  implementation  has  been  verified,  the  team 

Description  turns  to  characterizing  the  implementation  of  model  practices.  For  each  model 

practice,  and  each  instance  sampled  (i.e.,  in  each  basic  unit  or  support  function 
supplying  data  for  that  practice),  the  team  will  document  a  characterization  of  the 
extent  to  which  the  model  practice  (or  an  acceptable  alternative)  has  been 
implemented.  The  implementation-level  characterizations  are  then  aggregated  to 
the  organizational  unit  level. 

Characterizations  of  practice  implementation  are  used  as  a  means  to  focus 
appraisal  team  effort  on  areas  where  professional  judgment  is  needed,  and  to  aid 
in  reaching  team  consensus  on  the  extent  to  which  practices  are  implemented. 

In  preparation  for  validating  the  verified  information,  the  appraisal  team 
generates  preliminary  findings  that  summarize  potential  strengths  or  weaknesses 
that  support  judgments  about  practice  implementation.  The  preliminary  findings 
are  written  in  reference  to  a  single  model  practice,  and  are  abstracted  to  the  level 
of  the  organizational  unit,  unless  the  appraisal  is  planned  to  allow  for  basic  unit- 
level  preliminary  findings. 


Required  The  appraisal  team  shall 

Practices 

•  characterize  the  extent  to  which  appraisal  reference  model  practices  are 
implemented 

•  aggregate  practice  implementation  characterization  values  to  the 
organizational  unit  level 

•  document  summary  level  weaknesses  in  practice  implementation  if 
appropriate 

•  document  summary  level  strengths  in  practice  implementation,  if  appropriate 

•  generate  and  verify  preliminary  strengths  and  weaknesses 


Parameters 
and  Limits 

Characterize 

Implementation 


Characterization  of  the  implementation  of  each  practice  in  each  basic  unit  or 
each  support  function  is  assigned  according  to  the  table  below.  These  initial 
characterizations  may  be  assigned  through  consensus  of  a  mini-team  (consisting 
of  more  than  one  team  member),  or  may  be  based  on  the  consensus  of  the  entire 
appraisal  team. 


Judgments  described  in  the  table  below  are  contingent  on  the  appraisal  scope 
and  requirements  for  sufficient  data,  as  defined  in  1 . 1 .4,  Determine  Appraisal 
Scope  and  2.4.1,  Verify  Objective  Evidence.  In  some  cases,  only  artifacts  will 
be  examined,  and  in  some  cases  only  affirmations  will  be  examined. 
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2.4.2  Characterize  Implementation  of  Model  Practices  and  Generate 
Preliminary  Findings 


Parameters 
and  Limits 
(continued) 


Label 

Meaning 

Fully 

Implemented  (FI) 

Sufficient  artifacts  and/or  affirmations  are  present  (per 
1.1.4,  Determine  Appraisal  Scope  and  2.4.1,  Verify 
Objective  Evidence)  and  judged  to  be  adequate  to 
demonstrate  practice  implementation,  and  no 
weaknesses  are  noted. 

Largely 

Implemented  (LI) 

Sufficient  artifacts  and/or  affirmations  are  present  (per 

1.1.4  and  2.4.1)  and  judged  to  be  adequate  to 
demonstrate  practice  implementation,  and  one  or  more 
weaknesses  are  noted. 

Partially 

Implemented  (PI) 

Some  or  all  data  required  (per  1.1.4,  Determine  Appraisal 
Scope  and  2.4.1,  Verify  Objective  Evidence)  are  absent 
or  judged  to  be  inadequate,  some  data  are  present  to 
suggest  some  aspects  of  the  practice  are  implemented, 
and  one  or  more  weaknesses  are  noted. 

OR 

Data  supplied  to  the  team  (artifacts  and/or  affirmations) 
conflict  -some  data  indicate  the  practice  is  implemented 
and  some  data  indicate  the  practice  is  not  implemented, 
and  one  or  more  weaknesses  are  noted. 

Not 

Implemented  (Nl) 

Some  or  all  data  required  (per  1.1.4,  Determine  Appraisal 
Scope  and  2.4.1,  Verify  Objective  Evidence)  are  absent 
or  judged  to  be  inadequate,  data  supplied  does  not 
support  the  conclusion  that  the  practice  is  implemented, 
and  one  or  more  weaknesses  are  noted. 

Not  Yet  (NY) 

The  basic  unit  or  support  function  has  not  yet  reached  the 
stage  in  the  sequence  of  work,  or  point  in  time  to  have 
implemented  the  practice. 
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2.4.2  Characterize  Implementation  of  Model  Practices  and  Generate 
Preliminary  Findings 

The  table  below  summarizes  rules  for  aggregating  implementation-level 
characterizations  to  derive  organizational  unit-level  characterizations.  Consensus 
of  all  members  of  the  appraisal  team  is  necessary  for  organizational  unit-level 
characterizations. 


Parameters 
and  Limits 
(continued) 


The  column  labeled  “Implementations”  is  the  input  condition — the  pattern  of 
practice  implementation  characterizations  for  basic  units  or  support  functions. 
The  column  labeled  “Outcome”  is  the  resultant  aggregated  practice 
implementation  characterization  at  the  organizational  unit  level. 


Implementations 

Out¬ 

come 

Remarks 

All  FI  or  NY,  with  at 
least  one  FI 

FI 

All  implementations  are  characterized  FI  or  NY, 
with  at  least  one  FI. 

All  LI  or  FI  or  NY,  with 
at  least  one  LI 

LI 

All  implementations  are  characterized  LI  or  FI  or 

NY,  with  at  least  one  LI. 

At  least  one  LI  or  FI 

and  at  least  one  PI  or 

Nl 

LI  or  PI 

There  is  at  least  one  implementation  that  is 
characterized  as  LI  or  FI  and  at  least  one 
implementation  that  is  characterized  as  PI  or  Nl. 
Team  judgment  is  applied  to  choose  LI  or  PI  based 
on  whether  the  weaknesses,  in  aggregate,  have  a 
significant  negative  impact  on  goal  achievement. 

All  PI  orNI  or  NY, 
with  at  least  one  PI 

PI 

All  implementations  are  characterized  PI  or  Nl  or 

NY,  with  at  least  one  PI. 

All  Nl  or  NY,  with  at 
least  one  Nl 

Nl 

All  implementations  are  characterized  Nl  or  NY, 
with  at  least  one  Nl. 

All  NY 

NY 

All  implementations  are  characterized  NY.  There 
are  no  basic  units  or  support  functions  within  the 
organizational  unit  that  have  yet  reached  the  stage 
in  the  sequence  of  work  to  have  implemented  the 
practice.  (Note:  If  literally  all  basic  units  and  support 
functions  in  an  organizational  unit  have  not  reached 
the  stage  in  the  sequence  of  work  to  have 
implemented  the  practice,  but  will  in  the  future,  no 
rating  can  be  given  for  the  associated  goal.) 
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2.4.2  Characterize  Implementation  of  Model  Practices  and  Generate 
Preliminary  Findings 


Parameters 
and  Limits 
(continued) 

Generate 

Preliminary 

Findings 


For  any  practice  that  is  characterized  other  than  Fully  Implemented,  there  must 
be  a  statement  explaining  the  gap  between  what  the  organization  does  and  what 
the  model  expects. 

Statements  of  strengths  and  weaknesses  presented  to  the  organizational  unit  in 
the  form  of  preliminary  findings  for  validation  must  be  free  of  references  to 
specific  individuals,  basic  units,  or  support  functions.  Attribution  to  basic  units 
and  support  functions  is  permitted  as  a  tailoring  option  —  if  it  is  planned  and 
communicated  to  appraisal  participants  in  advance  of  the  data  collection 
activities.  However,  attribution  to  individuals  shall  never  be  permitted. 


Findings  (i.e.,  strengths  and  weaknesses)  must  be  verified,  that  is,  they  must  be 
based  on  sufficient  objective  evidence  (refer  to  activity  2.4.1,  Verify  Objective 
Evidence)  and  they  must  be  consistent  with  other  verified  findings.  Verified 
findings  cannot  be  both  true  and  mutually  inconsistent;  in  aggregate,  they 
constitute  a  set  of  truths  about  the  organizational  unit  that  must  be  consistent. 


Strengths  are  only  documented  if  the  implementation  of  a  practice  is  exemplary 
(above  and  beyond  the  capability  described  in  the  model),  and  reflects  a  strong 
asset  of  the  process  in  use.  An  adequate  implementation  of  a  model  practice  is 
not  a  strength.  Team  members  use  their  collective  experience  and  judgment  to 
determine  whether  or  not  they  have  uncovered  a  strength  to  highlight  in  the 
appraisal  findings. 
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2.4.2  Characterize  Implementation  of  Model  Practices  and  Generate 
Preliminary  Findings 


Implementation  a  weakness  is  defined  in  the  glossary  as  “the  ineffective,  or  lack  of, 


Guidance 

implementation  of  one  or  more  reference  model  practices.”  If  there  is  no  impact 
on  the  goal,  there  is  no  need  to  document  a  weakness.  If  the  appraisal  team 
identifies  a  process  improvement  suggestion  with  a  model  practice  that  is  not  an 
ineffective  (or  lack  of)  implementation  of  a  model  practice,  it  is  recorded  as  a 
note  rather  than  a  weakness. 

See  Appendix  B,  Alternative  Practice  Identification  and  Characterization 
Guidance,  for  information  on  applying  these  characterization  rules  in  situations 
where  acceptable  alternative  practices  have  been  identified. 

When  the  team  is  ready  to  perform  the  ratings,  these  characterizations  serve  to 
simplify  the  judgments.  The  team  is  then  able  to  focus  on  the  aggregation  of 
weaknesses  observed  to  determine  the  goal  satisfaction  ratings  (explained  in 
process  2.6,  Generate  Appraisal  Results).  Instantiations  in  situations  where  the 
basic  unit  or  support  function  has  not  yet  reached  the  appropriate  stage  in  the 
sequence  of  work  where  the  practice  would  be  enacted  are  characterized  as  Not 
Yet  (NY).  The  appraisal-planning  activities  are  expected  to  prevent  situations 
that  severely  limit  the  examples  of  actual  implementation  for  any  given 
practice. 

The  characterization  activity  can  begin  as  soon  as  sufficient  data  are  available. 

It  is  not  necessary  that  data  for  the  entire  organizational  unit  scope  be  available 
before  any  given  practice  can  be  characterized  at  the  implementation  level. 
However,  before  the  implementation  of  a  practice  across  the  organizational  unit 
can  be  characterized,  the  implementation-level  characterizations  have  been 
completed.  Each  instance  of  practice  enactment  is  characterized  using  the 
implementation-level  characterization  scheme. 

The  characterization  of  practice  implementation  for  the  organizational  unit  is 
carried  out  using  the  aggregation  rules  summarized  in  the  table  above.  These 
rules  provide  a  basis  for  identifying  the  areas  where  professional  judgment  is 
required,  and  simplify  the  areas  where  the  data  are  unanimous. 

♦ 
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2.5  Validate  Preliminary  Findings 


Purpose 

Validate  preliminary  findings,  including  weaknesses  (i.e.,  gaps  in  practice 
implementation)  and  strengths  (i.e.,  exemplary  implementation  of  model 
practices)  with  members  of  the  organizational  unit. 

Entry  Criteria 

Strengths  and  weaknesses  in  the  implementation  of  model  practices  have  been 
identified,  and  the  team  has  characterized  the  extent  to  which  model  practices  (or 
acceptable  alternatives  to  those  practices)  have  been  implemented.  Preliminary 
findings  at  the  level  of  the  organizational  unit  have  been  crafted  and  verified  for 
validation. 

Inputs 

•  appraisal  plan,  including  a  schedule  and  participants  for  data  validation 
activities 

•  strength/weakness  statements 

•  verified  objective  evidence 

•  characterizations  of  model  practice  implementation 

Activities 

2.5.1  Validate  Preliminary  Findings 

Outputs 

•  validated  appraisal  findings 

Outcome 

The  team’s  confidence  in  the  material  that  will  form  the  basis  for  appraisal 
outputs  is  increased,  and  the  process  of  transferring  ownership  of  these  results 
has  been  started.  Any  critical  deficiencies  in  the  data  on  hand  have  been 
identified  and  actions  to  resolve  these  issues  have  been  initiated. 

Exit  Criteria 

Preliminary  findings  have  been  validated  with  members  of  the  organization  who 
provided  appraisal  data. 

Key  Points 

This  activity  has  one  purpose — ensuring  the  validity  of  the  appraisal  data  and 
associated  outputs.  Managing  the  interaction  with  people  outside  of  the  team  is  a 
vitally  important  process  to  ensure  that  the  results  will  be  accurate. 

Tools  and 
Techniques 

Techniques  to  enhance  the  credibility  of  the  preliminary  findings  are  important. 
Using  a  flip  chart  or  note  taker  during  the  presentation  of  preliminary  findings  is 
often  effective  for  instilling  confidence  among  audience  members. 
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2.5  Validate  Preliminary  Findings 


Metrics 

Planned  versus  actual  effort  expended  for  this  process  (as  with  all  activities) 
will  assist  in  monitoring  progress  as  well  as  planning  subsequent  appraisals. 
Gauging  the  level  of  acceptance  for  preliminary  findings  can  be  facilitated  by 
computing  the  percentage  of  findings  adjusted  based  on  feedback,  then 
comparing  this  value  with  past  experience. 

Verification  and  The  attendees  of  preliminary  findings  presentations  are  likely  to  express 


Validation 

agreement  and/or  discuss  issues  with  the  data  being  validated.  The  appraisal 
team  leader  must  ensure  active  participation  in  these  activities  as  a  way  of 
confirming  that  the  verification  and  validation  process  is  working  as  intended. 
The  actions  taken  following  the  appraisal  will  provide  feedback  to  help  validate 
that  this  activity  was  successful. 

Records 

Characterizations  of  practice  implementation,  strength/weakness  statements, 
and  changes  made  based  on  feedback  will  be  recorded  for  subsequent  use  by  the 
team. 

Interfaces 
with  Other 
Processes 

During  the  conduct  of  an  appraisal,  the  team  gathers  and  analyzes  a  great  deal 
of  detailed  information.  Processes  described  earlier  in  this  document  clarify 
how  data  are  gathered  and  examined.  The  process  described  here  is  focused  on 
ensuring  the  data  reflects  actual  conditions  in  the  organization.  The  processes 
described  after  this  one  are  focused  making  reliable  and  valid  rating  judgments 
based  on  the  validated  data. 

Summary  of 
Activities 

When  team  members  have  achieved  their  planned  coverage  of  data  collection, 
the  preliminary  findings  are  validated  with  the  members  of  the  organization. 

This  final  activity  prior  to  rating  allows  team  members  to  build  confidence  that 
their  investigation  has  been  thorough,  and  the  members  of  the  organization  are 
provided  with  an  opportunity  to  correct  any  perceived  errors  in  the  appraisal 
data. 

♦ 
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2.5.1  Validate  Preliminary  Findings 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Validation  of  preliminary  findings  is  primarily  a  data  collection  activity,  and  the 
intent  is  to  validate  the  appraisal  team’s  understanding  of  the  processes 
implemented  within  the  organizational  unit.  Feedback  from  participants  may 
result  in  modifications  to  the  appraisal  team’s  inventory  of  objective  evidence. 
The  results  of  the  validation  activity  are  considered  in  the  fonnulation  of  final 
findings  and  goal  ratings.  These  latter  activities  cannot  commence  until  after  the 
validation  activity  has  occurred. 


The  appraisal  team  shall 

•  validate  preliminary  findings  (i.e.,  documented  strengths  and  weaknesses) 
with  members  of  the  organizational  unit 

Every  model  practice  characterized  as  Not  Implemented,  Partially  Implemented, 
or  Largely  Implemented  at  the  organizational  unit  level,  must  have  at  least  one 
weakness  associated  with  it. 

At  least  one  appraisal  participant  from  each  basic  unit  or  support  function 
providing  objective  evidence  and  from  any  associated  staff  function  must 
participate  in  the  set  of  validation  activities. 

Only  appraisal  participants  may  participate  (i.e.,  only  people  who  provided  data 
may  participate  in  validation). 

The  minimum  number  of  validation  sessions  required  is  one. 

At  the  Appraisal  Team  Lead’s  discretion,  or  at  the  request  of  the  sponsor, 
weaknesses  can  be  written  to  describe  connections  between  the  weakness  and 
business  goals. 

The  rules  of  confidentiality  and  the  expected  use  of  appraisal  data  must  be 
communicated  to  participants  in  each  validation  activity. 
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2.5.1  Validate  Preliminary  Findings 


Implementation 

Guidance 


Areas  where  the  appraisal  team’s  inventory  of  objective  evidence  is  insufficient 
to  satisfy  the  corroboration  (see  1.1.4,  Determine  Appraisal  Scope)  may  instead 
be  addressed  by  requests  for  additional  information  needed. 

Preliminary  findings  are  the  building  blocks  that  lead  to  the  judgment  of  goal 
satisfaction,  and  are  the  detailed  information  that  forms  the  basis  for  the  final 
findings.  As  an  intermediate  artifact  of  the  appraisal  process,  preliminary 
findings  are  used  to  ensure  traceability  between  inputs  to  the  appraisal  and 
outputs  of  the  appraisal. 

Feedback  from  participants  on  the  preliminary  findings  should  be  solicited  by  the 
appraisal  team  and  considered  for  possible  revisions  to  its  inventory  of  objective 
evidence. 

If  virtual  methods  such  as  video  conferences,  teleconferences,  and  other  similar 
technology  are  used  for  the  validation  session(s),  the  appraisal  team  leader 
should  ensure  that  these  methods  in  no  way  compromise  the  integrity  of  the 
appraisal  or  the  accuracy  of  the  results.  Virtual  methods  should  allow  for 
adequate  interaction  between  the  appraisal  team  members  and  the  appraisal 
participants  and  should  provide  mechanisms  for  the  appraisal  team  to  control  the 
validation  session.  Appropriate  arrangements,  such  as  meeting  rooms  at  remote 
sites,  should  be  made  and  checks  done  to  ensure  only  appraisal  participants 
attend  this  presentation  to  maintain  confidentiality. 

It  is  not  expected  that  preliminary  findings  will  provide  a  detailed  listing  of  the 
implementation  status  of  every  model  practice  in  every  sampled  basic  unit  or 
support  function.  Furthermore,  it  is  not  expected  that  the  preliminary  findings 
will  identify  the  status  of  individual  basic  units  or  support  functions  with  regard 
to  practice  implementation  or  goal  achievement.  An  appraisal  sponsor  may 
request  these  more  detailed  appraisal  results.  The  appraisal  team  leader  should 
negotiate  for  the  proper  allocation  of  time  to  accommodate  this  tailoring  option, 
and  the  expectation  that  such  information  will  be  preserved  at  the  end  of  the 
appraisal  should  be  made  clear  to  all  appraisal  participants. 
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2.5.1  Validate  Preliminary  Findings 


Implementation 

Guidance 

Preliminary 

Findings 

Presentations 


Implementation 

Guidance 

Focus  Groups 


An  interactive  presentation  is  the  most  effective  mechanism  for  validating  the 
preliminary  findings.  The  members  of  the  organization  who  provided  data  to 
the  appraisal  team  are  typically  brought  together  in  a  conference  room,  and  a 
slide  presentation  is  used  to  review  the  preliminary  findings  in  an  effort  to 
invite  people  to  provide  additional  data  or  express  their  agreement  with  the 
summary  statements.  The  audience  is  often  grouped  by  seniority  in  the 
organization,  and  separate  presentations  are  made  for  practitioners,  project 
managers,  and  middle  managers. 

During  the  presentation,  one  or  more  members  of  the  team  review  the 
preliminary  findings  statements  and  provide  the  audience  with  an  opportunity 
to  comment  or  ask  questions.  The  presenter  uses  only  the  words  crafted  by  the 
appraisal  team  and  avoids  elaborating  on  the  findings  using  his/her  own  words. 

When  questions  are  asked  about  a  preliminary  finding,  the  appraisal  team 
leader  provides  any  clarification  needed  to  understand  what  the  statement 
means.  However,  team  members  avoid  the  appearance  that  they  are  justifying 
the  content  of  the  statement. 

The  detailed  data  that  led  to  the  preliminary  findings  must  be  protected,  and 
negotiations  for  wording  changes  avoided.  The  appraisal  team  records  new  data 
made  available  to  them  without  commenting  on  how  the  data  may  be 
interpreted  or  how  the  findings  may  need  to  change. 


As  an  alternative  (or  in  addition)  to  the  presentation,  focus  groups  can  be  used 
to  probe  more  deeply  into  specific  areas  of  the  reference  model  with  a  targeted 
audience.  This  use  of  focus  groups  permits  the  team  to  explore  a  particular  area 
in  more  depth  to  help  sharpen  the  appraisal  results,  or  to  raise  the  visibility  of 
the  results  to  people  who  are  most  informed  on  the  topic.  For  example,  a  focus 
group  conducted  with  project  managers  could  be  an  ideal  environment  to 
validate  (and  gather  more  detailed  data  on)  the  topic  of  project  planning  and 
project  monitoring. 

In  contrast,  a  focus  group  composed  of  Engineering  Process  Group  (EPG) 
members  may  be  an  ideal  setting  to  validate  findings  associated  with  the 
organization’s  infrastructure  for  process  improvement.  The  preliminary 
findings  that  relate  to  the  group  may  be  distributed  as  handouts  or  displayed 
using  a  projector,  and  the  participants  can  engage  in  a  free-form  dialogue  with 
the  team  and  amongst  themselves.  Notes  taken  by  the  members  of  the  team  are 
treated  as  any  data  collected  during  an  interview  would  be. 
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2.5.1  Validate  Preliminary  Findings 


Implementation  Finally,  a  survey  instrument  can  be  used  in  addition  (or  as  an  alternative)  to 


Guidance 

either  of  the  techniques  above.  A  carefully  worded  instrument  that  asks 

Survey 

Instrument 

respondents  to  rate  their  level  of  agreement  with  the  finding  statement,  and 
provides  an  opportunity  for  written  feedback,  can  provide  a  low-cost  and  timely 
source  of  data  for  the  team. 

♦ 


CMU/SEI-201 1-HB-001 


154 


2.6  Generate  Appraisal  Results 


Purpose 

Rate  goal  satisfaction  based  on  the  extent  of  practice  implementation  throughout 
the  organizational  scope  of  the  appraisal.  The  extent  of  practice  implementation 
is  judged  based  on  validated  data  (e.g.,  artifact  and  affirmation  objective 
evidence)  collected  from  the  entire  representative  sample  of  the  organizational 
unit.  Aggregate  ratings  (process  area  ratings,  maturity  level  ratings,  capability 
ratings,  etc.)  are  driven  by  the  goal  satisfaction  ratings. 

Entry  Criteria 

The  set  of  validated  preliminary  findings,  statements  of  practice  implementation 
strengths  and  weaknesses,  and/or  tabulations  of  validated  objective  evidence  of 
practice  implementation  on  which  they  are  based  are  available.  Team  members 
are  confident  that  they  have  all  the  data  needed  to  make  rating  judgments.  The 
data  obtained  completely  covers  the  practices  within  the  defined  appraisal 
reference  model  scope  and  the  entire  representative  sample  selected  for  the 
organizational  unit. 

Inputs 

appraisal  data 

•  validated  preliminary  findings 

•  tabulations  of  objective  evidence  of  practice  implementation 

•  annotated  worksheets,  checklists,  working  notes 

Activities 

2.6.1  Derive  Findings  and  Rate  Goals 

2.6.2  Determine  Process  Area  Ratings 

2.6.3  Determine  Process  Area  Profile 

2.6.4  Determine  Maturity  Level 

2.6.5  Document  Appraisal  Results 

Outputs 

•  final  findings 

•  recorded  rating  decisions 

Outcome 

A  formal  rating  decision  for  each  appraisal  reference  model  component  that  was 
planned  to  be  rated,  and  for  which  the  team  obtained  complete  or  sufficient  data 

Exit  Criteria 

Ratings  against  all  components  per  the  plan  have  been  made  and  recorded. 
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2.6  Generate  Appraisal  Results 


Key  Points 

The  judgment  of  goal  satisfaction  is  based  on  and  traceable  to  the  extent  of  the 
implementation  of  practices  associated  with  that  goal  (or  alternative  practices 
contributing  equivalently  to  goal  satisfaction). 

Success  in  this  activity  is  driven  by  team  members’  ability  to  limit  their  focus 
to  the  data  that  support  the  judgments,  and  to  avoid  issues  that  threaten  their 
ability  to  be  objective.  This  activity  can  create  a  great  deal  of  stress  for  team 
members  under  pressure  to  help  their  organization  “do  well.”  The  appraisal 
team  leader  must  skillfully  facilitate  this  activity  when  external  pressures  exist. 

Tools  and 
Techniques 

There  is  a  significant  amount  of  data  to  review  in  making  each  round  of 
judgments.  Rating  worksheets  and  automated  support  tools  facilitate  the 
team’s  decision-making  process  by  presenting  necessary  data  in  a  concise, 
well-organized  manner.  When  controversial  issues  are  encountered,  the 
appraisal  team  leader  must  actively  facilitate  to  ensure  that  the  team  remains 
focused  on  the  pertinent  issues.  Strategic  rest  breaks,  and  sequencing  and 
pacing  critical  discussions,  are  often  keys  to  success. 

Metrics 

•  planned  versus  actual  effort  for  each  component  rated 

•  number  of  model  components  rated  satisfied  or  unsatisfied 

Verification  and  The  appraisal  team  leader  verifies  that  the  rating  process  was  performed  in 


Validation 

accordance  with  the  method  rules  and  the  rating  baseline  selected  and 
documented  in  the  appraisal  plan.  Work  aids  used  to  record  the  team 
judgments  help  ensure  traceability  to  the  basis  for  the  rating  judgments. 

Records 

A  worksheet  or  other  work  aid  may  be  used  to  make  a  record  of  the  rating 
decisions.  A  process  area  profile  is  often  an  effective  means  of  recording  and 
communicating  these  results. 
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2.6  Generate  Appraisal  Results 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


The  rating  judgments  made  by  the  appraisal  team  are  dependent  on  the  quality 
of  the  data  available  to  them,  as  well  as  their  ability  to  reliably  judge  the 
implementation  and  institutionalization  of  practices  in  the  organization  that 
relate  to  the  reference  model.  All  processes  previously  described  contribute  to 
the  team’s  ability  to  effectively  execute  this  process  to  generate  appraisal 
results. 

The  Analyze  Requirements  process  establishes  the  rating  baseline,  the 
organizational  unit  to  which  ratings  will  apply,  and  the  purpose  for  which  the 
ratings  are  used.  The  Develop  Appraisal  Plan  process,  in  conjunction  with  the 
Obtain  and  Inventory  Initial  Objective  Evidence  and  Prepare  for  Appraisal 
Conduct  processes,  determines  the  sample  of  the  organizational  unit  for  which 
data  will  be  collected  and  from  which  the  ratings  will  be  determined. 

The  Select  and  Prepare  Team  process  ensures  that  the  team  has  sufficient 
knowledge  and  skills  to  interpret  the  data  and  arrive  at  sound  rating  judgments. 
The  Examine  Objective  Evidence  and  Document  Objective  Evidence  processes 
provide  the  basic  information  that  is  needed  to  support  judgments  in  a  form  that 
facilitates  making  the  judgments. 

The  Verify  Objective  Evidence  process  characterizes  the  extent  to  which  the 
organizational  unit  implements  practices  in  the  model  (or  acceptable 
alternatives).  The  Validate  Preliminary  Findings  process  validates  findings  that 
describe  any  weaknesses  or  strengths  associated  with  the  practice 
implementations. 

Upon  the  successful  execution  of  these  processes,  the  team  is  ready  to  rate  the 
satisfaction  of  goals  dependent  on  those  practices. 


The  required  and  fundamental  rating  activity  involves  making  team  judgments 
about  goal  satisfaction  for  each  and  every  goal  within  the  appraisal  reference 
model  scope.  Once  goal  satisfaction  has  been  detennined,  optional  aggregate 
ratings  may  be  produced. 

The  first  level  of  aggregate  rating  aggregates  goal  satisfaction  to  process  area 
ratings.  This  process  is  described  in  Section  2.6.2.  The  second  level  of 
aggregate  rating  is  to  produce  a  process  area  profile,  as  described  in  Section 
2.6.3.  The  third  and  final  level  of  aggregate  rating  is  to  produce  a  maturity  level 
rating  for  the  entire  organizational  unit. 


♦ 
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2.6.1  Derive  Findings  and  Rate  Goals 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  judgments  made  about  goal  satisfaction  are  driven  by  the  preliminary 
findings  that  were  documented  by  the  appraisal  team  and  validated  by  appraisal 
participants  as  well  as  the  extent  of  implementation  of  associated  practices.  The 
preliminary  findings  focus  on  exemplary  practice  implementations  as  well  as 
gaps  in  the  implementation  of  practices.  When  perfonning  goal  ratings,  the  team 
judges  whether  or  not  these  weaknesses  in  the  implementation  of  practices  (in 
aggregate)  threaten  the  organizational  unit’s  ability  to  satisfy  the  goals  associated 
with  the  practices. 


The  appraisal  team  shall  do  the  following: 

•  derive  final  findings  using  preliminary  findings  statements,  feedback  from 
validation  activities,  and  any  additional  objective  evidence  collected  as  a 
result  of  the  validation  activities 

•  rate  each  goal  within  the  reference  model  scope  of  the  appraisal,  based  on 
the  practice  implementation  characterizations  at  the  organizational  unit  level 
as  well  as  the  aggregation  of  weaknesses  associated  with  that  goal 

•  obtain  appraisal  team  consensus  on  the  findings  statements  and  ratings 
generated  for  the  organizational  unit  level 


When  deriving  final  findings,  the  aim  is  to  create  goal-level  statements  that 
summarize  the  strengths  and  weaknesses  in  practice  implementation.  These 
statements  must  be  abstracted  to  the  level  of  the  organizational  unit,  and  cannot 
focus  on  individual  basic  unit  or  increments  (unless  the  tailoring  option  for  basic 
unit-  or  increment-specific  findings  has  been  agreed  on  during  planning). 

A  goal  must  be  rated  Not  Rated  if  there  are  any  associated  practices  that  are  not 
characterized  at  the  organizational  unit  level  or  that  are  characterized  as  Not  Yet 
at  the  organizational  unit  level. 

A  goal  is  rated  Not  Rated  if  the  associated  set  of  objective  evidence  does  not 
meet  the  defined  criteria  for  sufficient  data  coverage. 

The  goal  is  rated  Satisfied  if  and  only  if  both  of  the  following  are  true: 

•  All  associated  practices  are  characterized  at  the  organizational  unit  level  as 
either  Largely  Implemented  or  Fully  Implemented. 

•  The  aggregation  of  weaknesses  associated  with  the  goal  does  not  have  a 
significant  negative  impact  on  goal  achievement. 

For  a  goal  to  be  rated  as  Unsatisfied,  the  team  must  be  able  to  describe  how  the 
set  of  documented  weaknesses  (or  single  weakness)  led  to  this  rating. 
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2.6.1  Derive  Findings  and  Rate  Goals 


Implementation 

Guidance 


Any  endeavor  that  results  in  producing  a  score,  grade,  or  rating  is  by  definition 
an  area  of  sensitivity  to  those  affected  by  the  outcome.  An  objective  and  clear 
basis  for  assigning  a  rating  lessens  this  sensitivity  and  results  in  a  more 
consistent  basis  of  comparison  among  the  organizational  units  and  goals  rated. 

Judgments  made  prior  to  and  during  the  rating  process  should  be  based  on 
observable  facts  and  should  be  made  at  the  lowest  level  of  abstraction  that 
makes  sense.  In  the  case  of  SCAMPI  A  appraisals,  the  lowest  level  of 
abstraction  is  characterizing  the  extent  of  practice  implementation  for  each 
basic  unit  or  support  function  within  the  representative  sample. 

Characterizations  made  at  the  instantiation  level  are  aggregated  into  a 
characterization  of  the  extent  of  practice  implementation  throughout  the 
organizational  unit,  as  described  earlier  in  process  2.4,  Verify  Objective 
Evidence.  The  judgment  of  goal  satisfaction  is  then  based  on,  and  directly 
traceable  to,  the  extent  of  implementation  of  practices  associated  with  that  goal. 
(See  Appendix  B,  Alternative  Practice  Identification  and  Characterization 
Guidance,  for  information  regarding  the  judgment  of  goal  satisfaction  when 
acceptable  alternative  practices  are  involved.) 

Findings  should  be  phrased  in  tenns  that  best  support  the  appraisal  sponsor’s 
decision  making  and  taking  and  taking  action  to  address  appraisal  results. 


♦ 
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2.6.2  Determine  Process  Area  Ratings 


Activity 

Description 


Required 

Practices 


The  appraisal  team  may  produce  ratings  for  process  areas.  Assigning  process 
area  ratings  is  an  optional  activity  that  is  selected  at  the  discretion  of  the 
appraisal  sponsor  and  documented  in  the  appraisal  plan.  Depending  on  the 
reference  model  chosen,  the  process  area  ratings  may  reflect  a  staged  architecture 
(satisfied  vs.  unsatisfied)  or  a  continuous  architecture  (capability  levels  0  to  3). 


If  process  area  ratings  are  selected  as  an  appraisal  output  by  the  sponsor,  the 

appraisal  team  shall  do  the  following: 

•  If  using  a  continuous  representation,  assign  a  capability  level  to  each  process 
area  within  the  scope  of  the  appraisal,  based  on  the  highest  level  for  which 
all  specific  goals  and  generic  goals  within  the  appraisal  scope  have  been 
satisfied.  (See  the  parameters  and  limits  section  that  follows  for  a  more 
specific  discussion.) 

•  If  using  a  staged  representation,  rate  the  satisfaction  of  each  process  area 
within  the  scope  of  the  appraisal,  based  on  the  satisfaction  ratings  assigned  to 
all  goals  included  in  that  process  area.  See  the  parameters  and  limits  section 
(below)  for  a  more  specific  discussion. 

•  If  any  of  the  goals  are  rated  Not  Rated  and  none  of  the  other  goals  are  rated 
Unsatisfied,  then  the  process  area  is  rated  Not  Rated. 

•  When  a  process  area  is  determined  to  be  outside  of  the  organizational  unit’s 
scope  of  work,  the  process  area  is  designated  as  “not  applicable”  and  is  not 
rated. 

•  When  an  applicable  process  area  is  outside  of  the  scope  of  the  model  used 
for  the  appraisal,  the  process  area  is  designated  as  “out  of  scope”  and  is  not 
rated. 
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2.6.2  Determine  Process  Area  Ratings 


Parameters 
and  Limits 

For  an  appraisal  using  a  continuous  representation,  the  following  table  defines 
the  basis  for  capability  level  ratings: 

Capability 

Level 

Process  Areas 

0 

Default  Rating 

1 

Generic  goal  for  capability  level  1  is  rated  Satisfied. 

(All  specific  goals  are  rated  Satisfied.) 

2 

Generic  goals  for  capability  levels  1  and  2  are  rated  Satisfied. 

(All  specific  goals  are  rated  Satisfied.) 

3 

Generic  goals  for  capability  levels  1 , 2,  and  3  are  rated  Satisfied. 

(All  specific  goals  are  rated  Satisfied.) 

For  an  appraisal  using  a  staged  representation  model,  the  “satisfied”  rating  for  a 
process  area  may  depend  on  the  target  maturity  level  for  the  appraisal  —  if 
perfonning  the  maturity  level  rating  was  selected  by  the  appraisal  sponsor. 

For  CMMI  models,  rating  a  process  area  “staged  at  maturity  level  2”  as  satisfied 
would  require  a  satisfied  rating  for  generic  goal  2  in  order  to  support  a  maturity 
level  2  outcome  for  the  appraisal.  If  the  target  maturity  level  for  the  appraisal  is 
level  3,  then  generic  goal  3  must  be  satisfied  in  order  for  the  process  area  to  be 
rated  as  satisfied. 

Implementation 

Guidance 

None 

♦ 
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2.6.3  Determine  Process  Area  Profile 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Implementation 

Guidance 


The  appraisal  team  may  create  a  process  area  profile  (called  “capability  level 
profile”  or  “achievement  profile”  in  CMMI  models)  that  graphically  depicts  the 
ratings  assigned  to  each  process  area  within  the  scope  of  the  appraisal.  The 
generation  of  a  process  area  profile  is  an  optional  activity,  selected  at  the 
discretion  of  the  appraisal  sponsor  and  documented  in  the  appraisal  plan. 


If  a  process  area  profile  was  selected  as  an  output  by  the  sponsor,  the  appraisal 
team  shall  generate  a  process  area  profile  depicting  the  ratings  for  each  process 
area  within  the  scope  of  the  appraisal. 


A  simple  bar  chart  can  be  used  for  the  display.  Each  process  area  is  represented 
in  a  single  bar  along  the  horizontal  axis,  and  the  vertical  axis  represents  the 
rating  dimension  (i.e.,  ‘Satisfied’  vs.  ‘Unsatisfied’  or  a  capability  level  of  0,  1,2 
or  3).  The  height  of  each  bar  communicates  the  rating  for  the  process  area 
represented. 

Capability  levels  take  only  the  values  0,  1,  2,  or  3.  Intermediate  values  (e.g., 

2.7)  are  not  defined  for  this  appraisal  outcome,  and  any  embellishment  of  the 
Capability  Profile  with  such  values  is  outside  the  boundaries  of  SCAMPI  A. 

Satisfaction  ratings,  which  may  take  on  one  of  two  values  “Satisfied”  or 
“Unsatisfied”  for  each  process  area,  are  used  when  the  appraisal  is  using  the 
staged  representation. 


Process  area  profiles  are  typically  used  to  communicate  the  rating  results  to  the 
sponsor  and  others  designated  by  the  sponsor. 

Comparing  different  process  areas  with  respect  to  their  satisfaction  status  or 
capability  level  may  be  informative  in  discussing  trends  or  patterns  in  the 
organization. 

This  activity  may  be  omitted  entirely:  it  is  a  tailoring  option. 


♦ 
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2.6.4  Determine  Maturity  Level 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Implementation 

Guidance 


Historically,  one  of  the  most  visible  outcomes  of  an  appraisal  has  been  the 
maturity  level  rating.  The  determination  of  a  maturity  level  rating  is 
straightforward  and  is  derived  mechanically  from  the  ratings  assigned  to 
process  areas.  Assigning  a  maturity  level  rating  is  an  optional  activity,  selected 
at  the  discretion  of  the  appraisal  sponsor  and  documented  in  the  appraisal  plan. 


If  a  maturity  level  rating  was  selected  as  an  output  by  the  sponsor,  the  appraisal 
team  shall  assign  the  maturity  level  based  on  the  ratings  assigned  to  process 
areas. 


When  using  the  staged  representation,  the  maturity  level  determined  is  the 
highest  level  at  which  all  process  areas  contained  within  the  maturity  level,  and 
within  all  lower  maturity  levels,  are  rated  as  Satisfied  or  Not  Applicable.  (Note: 
as  explained  in  the  parameters  and  limits  for  rating  processes  areas,  to  achieve 
maturity  level  3,  CMMI  models  also  require  satisfaction  of  generic  goal  3  for 
process  areas  staged  at  maturity  level  2. 

When  using  the  continuous  representation,  please  consult  the  discussion  of 
equivalent  staging  in  the  reference  model  for  an  explanation  of  the  capability 
level  profiles  required  for  each  maturity  level  rating. 

To  determine  a  maturity  level  as  an  output  of  the  appraisal,  the  model  scope  of 
the  appraisal  must  include  the  minimum  set  of  process  areas  required  by  the 
appraisal  reference  model. 


This  activity  may  be  omitted  entirely,  as  it  is  a  tailoring  option.  If  a  maturity 
level  is  to  be  reported,  the  process  area  ratings  that  form  the  basis  for  the 
maturity  level  rating  are  derived  as  described  in  activity  2.6.2,  Determine 
Process  Area  Ratings. 


♦ 
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2.6.5  Document  Appraisal  Results 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Implementation 

Guidance 


The  results  of  conducting  the  appraisal  are  documented  for  reporting.  Verbal 
reports  of  the  rating  outcomes  or  oral  explanations  of  implementation 
weaknesses  discovered  by  the  team  are  not  sufficient  to  communicate  appraisal 
results. 


The  appraisal  team  shall  document  the  following: 

•  document  the  final  findings 

•  document  the  rating  outcome(s) 

•  document  the  appraisal  disclosure  statement 


The  appraisal  disclosure  statement  and  the  set  of  appraisal  outputs  agreed  on 
with  the  appraisal  sponsor  must  be  documented. 

The  appraisal  disclosure  statement,  the  goal  ratings,  and  the  associated  findings 
must  be  documented  as  a  part  of  the  appraisal  information  returned  to  the  SEI. 


This  activity  is  focused  on  collecting  and  documenting  the  results  of  activities 
related  to  the  generation  of  findings  and  ratings.  Depending  on  who  will  receive 
the  results,  multiple  forms  of  the  results  may  be  needed.  Certain  data  may  not 
be  appropriate  for  all  audiences,  or  the  style  and  language  of  the  results  may 
need  to  be  adjusted  to  best  fit  the  needs  of  the  recipients. 

The  documented  appraisal  results  are  typically  provided  in  a  final  findings 
presentation,  described  in  activity  3.1.1,  Deliver  Final  Findings. 


♦ 
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3  Report  Results 

3.1  Deliver  Appraisal  Results 


Purpose 

Provide  credible  appraisal  results  that  can  be  used  to  guide  actions.  Represent  the 
strengths  and  weaknesses  of  the  processes  in  use  at  the  time.  Provide  ratings  (if 
planned  for)  that  accurately  reflect  the  capability  level  or  maturity  level  of  the 
processes  in  use. 

Entry  Criteria 

•  Objective  evidence  has  been  verified  (through  the  team  process). 

•  Preliminary  findings  have  been  validated. 

•  Ratings  have  been  determined  (for  model  components  selected  for  rating). 

•  Final  findings  have  been  created  and  reviewed  by  the  team. 

Inputs 

•  appraisal  data 

•  final  findings 

•  ratings 

•  appraisal  artifacts 

•  appraisal  plan 

Activities 

3.1.1  Deliver  Final  Findings 

3.1.2  Conduct  Executive  Session(s) 

3.1.3  Plan  for  Next  Steps 

Outputs 

•  documented  final  findings 

•  final  report  (if  requested) 

•  recommendations  report  (if  requested) 

•  process  improvement  action  plan  (if  requested) 

Outcome 

•  The  sponsor  and  the  appraised  organizational  unit  are  provided  with  the 
results  of  the  appraisal. 

•  A  valid  and  reliable  characterization  of  the  current  state  of  the  processes  in 
use  across  the  organizational  unit  is  documented. 

Exit  Criteria 

•  Appraisal  results  are  delivered  to  the  appraisal  sponsor  and  organizational 
unit. 

•  An  executive  session  is  conducted,  if  appropriate. 
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3.1  Deliver  Appraisal  Results 


Key  Points 

The  appraisal  results  are  intended  to  support  decision  making,  and  should  be 
delivered  in  a  way  that  promotes  appropriate  actions.  Whether  the  appraisal  was 
conducted  for  internal  process  improvement,  supplier  selection,  or  process 
monitoring  purposes,  the  delivery  of  results  should  facilitate  the  actions  that  will 
be  driven  by  the  information. 

Tools  and 
Techniques 

Templates  containing  standard  information  for  use  in  a  final  findings  briefing  are 
provided  to  all  SCAMPI  Lead  Appraisers.  Experienced  appraisal  team  leaders 
frequently  use  electronic  (database)  tools  that  support  the  transformation  of  raw 
appraisal  data  into  appraisal  results.  These  tools  may  be  useful  in  real  time  as 
appraisal  results  are  presented.  Strategies  for  presenting  and  packaging  the  results 
should  leverage  presentation  and  documentation  techniques  that  best  suit  the 
audience. 

Metrics 

It  is  highly  recommended  that  the  attendance  at  the  final  briefing  (if  one  is  held) 
be  recorded.  Significant  absenteeism  of  key  stakeholders  is  likely  to  be  an 
indication  of  risk  for  future  success  in  addressing  the  appraisal  findings. 

Verification  and  The  required  elements  of  appraisal  results  are  specified  in  the  activity  description 


Validation 

found  here,  and  a  checklist  can  support  verification  that  these  elements  are 
present.  Validation  of  this  activity  can  only  occur  after  the  appraisal  is  complete. 

Records 

•  final  findings 

•  final  report  (if  requested) 

•  recommendations  report  (if  requested) 

•  process  improvement  action  plan  (if  requested) 
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3.1  Deliver  Appraisal  Results 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


Upon  completion  of  the  Generate  Appraisal  Results  process,  the  ratings  and 
findings  generated  are  used  to  prepare  and  deliver  the  final  appraisal  results  to 
the  appraisal  sponsor  and  organizational  unit.  The  appraisal  results  become  part 
of  the  appraisal  record,  which  is  discussed  in  process  3.2,  Package  and  Archive 
Appraisal  Assets. 


The  final  findings  contain  the  validated  strengths,  weaknesses,  and  ratings  (as 
defined  by  the  appraisal  plan),  reflecting  the  organizational  process  capability 
and/or  maturity  level  for  process  areas  within  the  appraisal  scope.  Other 
appraisal  outputs,  as  requested  by  the  appraisal  sponsor  and  documented  in  the 
appraisal  plan,  are  generated  and  provided.  Optionally,  a  separate  executive 
session  may  also  be  held  to  clarify  and  discuss  the  appraisal  results  from  a 
senior  management  perspective  that  facilitates  decision  making.  Plans  are 
established  for  acting  on  the  appraisal  results. 


♦ 
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3.1.1  Deliver  Final  Findings 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  final  findings  contain  a  summary  of  the  documented  strengths  and 
weaknesses  for  each  process  area  within  the  appraisal  scope,  as  well  as  additional 
information  that  provides  context  for  the  findings.  The  generation  of  the  findings 
is  addressed  in  activity  2.6.1,  Derive  Findings  and  Rate  Goals;  this  activity 
relates  to  the  delivery  of  these  findings  to  the  appraisal  sponsor  and  appraised 
organization.  These  findings  may  be  in  a  summarized  form,  with  the  detailed 
findings  provided  as  backup  information,  and  is  often  presented  using  view 
graphs  in  a  meeting  room  or  auditorium. 

In  addition  to  the  final  findings,  a  draft  appraisal  disclosure  statement 
summarizing  the  results  of  the  appraisal  is  provided  to  the  appraisal  sponsor. 


The  appraisal  team  shall 

•  provide  appraisal  final  findings,  signed  by  the  appraisal  team  leader  and  all 
appraisal  team  members,  to  the  appraisal  sponsor 

•  provide  an  appraisal  disclosure  statement  to  the  appraisal  sponsor 
summarizing  the  appraisal  results  and  conditions  under  which  the  appraisal 
was  performed.  Use  the  appraisal  disclosure  statement  template  provided  by 
the  SEI.  The  appraisal  disclosure  statement  must  be  signed  by  the  appraisal 
team  leader  and  appraisal  sponsor. 


Required  elements  of  the  final  findings  include 

•  a  summary  of  the  appraisal  process 

•  the  findings  (i.e.,  summary  of  documented  strengths  and  weaknesses) 

•  signed  verification  that  the  appraisal  team  leader  and  all  appraisal  team 
members  agree  with  the  findings  and  any  ratings  reported 

Appraisal  team  consensus  must  be  obtained  on  the  wording  of  the  final  findings, 

to  ensure  that  the  whole  team  supports  the  accuracy  of  the  described  appraisal 

results. 

The  team,  when  delivering  the  final  findings,  must  adhere  to  some  important 

principles: 

•  If  a  model  component  is  reported  as  Unsatisfied,  the  corresponding  findings 
of  weaknesses  that  caused  the  team  to  make  that  judgment  must  also  be 
reported. 

•  Confidentiality  and  non-attribution  principles  apply  to  statements  made  in 
the  presentation  of  final  findings. 
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3.1.1  Deliver  Final  Findings 


Implementation 

Guidance 


The  appraisal  disclosure  statement  is  a  summary  statement  describing  the 
appraisal  results  that  includes  the  conditions  and  constraints  under  which  the 
appraisal  was  performed.  It  contains  information  considered  essential  to 
adequately  interpret  the  meaning  of  assigned  maturity  level  or  capability  level 
ratings.  The  appraisal  disclosure  statement  is  prepared  by  the  appraisal  team 
leader  and  provided  to  the  appraisal  sponsor. 

The  appraisal  disclosure  statement  is  considered  a  draft  at  this  stage  of  the 
appraisal  process,  in  that  the  it  also  contains  an  affirmation  that  all  appraisal 
requirements  have  been  satisfied,  which  cannot  be  claimed  until  the  completion 
of  all  appraisal  activities. 

A  template  for  a  final  findings  briefing,  describing  its  typical  contents  and 
format,  is  provided  to  SCAMPI  Lead  Appraisers  as  a  work  aid  by  the  SEI. 

Findings  include  a  summary  of  strengths  and  weaknesses  determined  for  each 
process  area  within  the  appraisal  reference  model  scope.  This  summary  may 
also  include  global  findings  that  apply  across  multiple  process  areas,  and  non¬ 
reference  model  findings  that  affect  the  implementation  (positively  or 
negatively)  of  associated  processes  within  the  organizational  unit. 

Normally,  the  appraisal  team  leader  presents  the  final  findings.  In  some 
applications  of  the  method  for  internal  process  improvement,  the  team  may 
elect  to  have  an  appraisal  team  member  from  the  organizational  unit  provide  the 
briefing  to  encourage  the  acceptance  of  the  final  findings  and  ownership  of  the 
appraisal  results  for  follow-on  action. 

As  a  courtesy,  the  appraisal  team  can  consider  informing  the  appraisal  sponsor 
and/or  the  senior  site  manager  of  the  appraisal  results  prior  to  presenting  them 
publicly  in  the  final  findings  briefing.  This  private  briefing  may  help  them 
avoid  surprises  and  obtain  feedback  on  ways  to  present  the  findings  that  best 
meet  the  needs  of  the  sponsor,  appraisal  participants,  and  the  organizational 
unit.  See  activity  3.1.2,  Conduct  Executive  Session(s)  for  a  description  of  topics 
for  discussion. 

If  virtual  methods,  such  as  video  conferences,  teleconferences,  or  other  similar 
technology  are  used  for  the  final  findings  presentation,  the  appraisal  team  leader 
should  ensure  that  these  methods  in  no  way  compromise  the  integrity  of  the 
appraisal. 

The  number  and  scope  of  findings  reported  will  affect  the  impact  of  appraisal 
results,  whether  or  not  the  team  intends  for  it  to  happen.  There  are  times  when 
providing  a  long  list  of  details  is  beneficial.  Other  times,  high-level  summaries 
are  more  appropriate. 


♦ 
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3.1.2  Conduct  Executive  Session(s) 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Implementation 

Guidance 


The  executive  session  is  an  optional  activity  that  may  be  performed  at  the 
discretion  of  the  appraisal  sponsor  or  senior  site  manager.  The  executive 

session  provides  the  appraisal  sponsor,  senior  site  manager,  and  invited  staff  a 
private  opportunity  to  (a)  discuss  with  the  appraisal  team  leader  any  issues  with 
the  appraisal,  (b)  obtain  clarification  of  the  appraisal  results,  (c)  confirm 
understanding  of  the  process  issues,  and  (d)  provide  guidance  regarding  focus, 
timing,  and  priorities  of  the  recommendations  report  and  follow-on  activities. 


If  the  option  is  selected,  hold  a  private  meeting  between  the  appraisal  team 
leader  and  the  sponsor,  including  any  participants  invited  by  the  sponsor. 


If  an  executive  session  is  conducted,  the  confidentiality  and  non-attribution  of 
data  sources  must  be  maintained. 


The  intent  of  the  executive  sessions  is  to  ensure  that  the  appraisal  sponsor 
and/or  the  senior  site  manager  have  a  sound  understanding  of  the  appraisal 
results.  Any  feedback  obtained  from  these  executive  sessions  should  be 
recorded.  All  rules  for  confidentiality  and  non-attribution  are  still  in  effect. 


♦ 
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3.1.3  Plan  for  Next  Steps 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Implementation 

Guidance 


Implementation 

Guidance 

Process 
Improvement 
Action  Planning 


Following  the  delivery  of  the  appraisal  results,  a  plan  for  follow-on  activities  is 
determined.  The  planned  follow-on  activities  are  typically  defined  in  the 
appraisal  plan,  reflecting  sponsor  requests  for  additional  appraisal  tasks  and 
products  necessary  to  meet  appraisal  objectives,  or  for  a  commitment  to  take 
action  on  the  appraisal  results.  Follow-on  activities  may  include 

•  development  of  a  final  report 

•  development  of  a  recommendations  report  or  briefing 

•  generation  or  update  of  a  process  improvement  plan 


None 


None 


Planning  for  next  steps  includes  optional  activities  such  as 

•  development  of  a  final  report  by  the  appraisal  team,  summarizing  the 
appraisal  results  for  delivery  to  the  appraisal  sponsor 

•  submission  of  appraisal  team  recommendations  report  for  action  on  the 
appraisal  findings 

•  generation  of  a  process  improvement  action  plan  for  the  organizational  unit 
to  act  on  the  appraisal  findings 

In  addition  to  specifying  the  activities  to  be  performed,  these  plans  usually 
include  the  assignment  of  responsibility,  schedule,  and  estimated  resources  for 
the  implementation  of  the  follow-on  actions.  The  plans  established  can  be  used  to 
track  the  progress  of  the  follow-on  activities  over  time. 


Findings  and  recommendations  from  the  appraisal  team  can  be  used  by  the 
organizational  unit  to  establish  action  plans  for  process  improvement.  This  use  of 
findings  and  recommendations  is  an  optional  output  most  often  used  in  internal 
process  improvement  or  process-monitoring  applications  of  the  appraisal 
method. 

Recommendations  reports  often  include  a  prioritized  list  of  improvement 
activities,  including  the  development  of  an  improvement  plan  that  defines  the 
tasks,  schedules,  and  resources  necessary  for  implementation. 

Follow-on  appraisals  are  usually  performed  to  verify  improvement  progress.  A 
follow-on  appraisal  might  include  a  combination  of  Class  A,  Class  B,  and  Class 
C  appraisals  (refer  to  the  ARC  for  additional  details). 
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3.1.3  Plan  for  Next  Steps 


Implementation 

Guidance 

Final  Report 


Implementation 

Guidance 

Recommendations 

Report 


The  purpose  of  the  final  report  is  to  provide  details  or  explanations  beyond 
what  was  contained  in  the  final  findings.  The  generation  of  an  appraisal  final 
report  is  an  optional  activity  that,  if  requested  by  the  appraisal  sponsor, 
documents  the  execution  of  the  appraisal,  contains  detailed  appraisal  findings, 
and  forms  a  basis  for  action  planning.  This  baseline  is  used  for  subsequent 
reports  and  follow-on  actions,  and  also  may  be  an  input  for  use  in  subsequent 
appraisals. 

Items  contained  or  referenced  in  the  final  report,  either  in  their  entirety  or  as  a 
subset,  might  include 

•  executive  summary  of  the  appraisal  process  and  results 

•  appraisal  plan  (see  process  1 .2,  Develop  Appraisal  Plan) 

•  appraisal  record  (see  process  3.2,  Package  and  Archive  Appraisal  Assets) 

The  final  report  should  be  completed  as  soon  after  the  appraisal  as  possible, 
preferably  within  four  weeks.  The  appraisal  team  leader  usually  generates  the 
final  report;  other  team  members  may  also  contribute. 

The  format  and  content  of  the  final  report  may  vary  according  to  its  intended 
use  by  the  appraisal  sponsor.  In  its  simplest  form,  this  final  report  could  be  a  set 
of  notes  annotated  to  the  final  findings,  elaborating  on  some  aspect  of  the 
findings  or  capturing  essential  comments  or  recommendations  from  the 
appraisal  team. 

If  requested  by  the  appraisal  sponsor,  appraisal  team  recommendations  for 
taking  action  on  the  appraisal  results  can  be  provided  in  a  report.  These 
recommendations  can  provide  momentum  to  the  appraisal  follow-up  by  serving 
as  a  link  between  the  appraisal  findings  and  subsequent  decision  making  or 
action  plans.  The  emphasis  of  these  recommendations  depends  on  the  appraisal 
sponsor’s  objectives  and  planned  use  of  the  appraisal  results,  as  defined  in  the 
appraisal  plan.  This  emphasis  can  vary  widely  based  on  the  context  in  which 
the  appraisal  method  is  applied  (i.e.,  internal  process  improvement,  supplier 
selection,  or  process  monitoring). 

The  recommendations  report  should  be  completed  as  soon  after  the  Conduct 
Appraisal  phase  as  possible.  Depending  on  the  nature,  complexity,  and  use  of 
the  recommendations,  this  report  may  take  as  long  as  two  months  to  produce. 

Rather  than  generate  a  separate  recommendations  report,  a  common  alternative 
is  to  include  these  recommendations  in  the  final  findings  or  final  report. 

It  is  important  to  consider  the  possibility  that  the  expertise  needed  for  making 
the  appropriate  recommendations  may  be  beyond  the  level  of  expertise 
reflected  on  the  team. 


♦ 
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3.2  Package  and  Archive  Appraisal  Assets 


Purpose 


Entry  Criteria 


inputs 


Activities 


Outputs 


Outcome 


Exit  Criteria 


Preserve  important  data  and  records  from  the  appraisal,  and  dispose  of  sensitive 
materials  in  an  appropriate  manner. 


•  The  appraisal  has  been  conducted. 

•  Results  have  been  delivered  to  the  sponsor. 

•  All  appropriate  data  have  been  collected  and  retained  during  the  appraisal. 

•  appraisal  data 

appraisal  plan 

final  findings 

objective  evidence 

signed  appraisal  disclosure  statement 

•  appraisal  team  artifacts 

notes 

documented  practice  implementation  gaps 
preliminary  findings 
objective  evidence 

3.2. 1  Collect  Lessons  Learned 

3.2.2  Generate  Appraisal  Record 

3.2.3  Provide  Appraisal  Feedback  to  the  SEI 

3.2.4  Archive  and/or  Dispose  of  Key  Artifacts 


•  appraisal  data  package 

•  appraisal  record 

•  completed  forms  and  checklists 

•  sanitized  data  (as  appropriate  and  agreed  upon  during  planning) 

•  lessons  learned  (appraisal  team,  organization) 

Data  and  artifacts  are  appropriately  archived  or  destroyed.  The  team  has  captured 
lessons  and  data  to  help  improve  the  appraisal  process.  Requirements  for 
providing  appraisal  artifacts  to  stakeholders  and  the  SEI  are  met. 


•  Appraisal  assets  are  baselined  and  archived. 

•  Required  reports  are  delivered  to  the  appropriate  stakeholders. 

•  Artifacts  containing  sensitive  information  are  disposed  of  in  an  appropriate 
manner. 
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3.2  Package  and  Archive  Appraisal  Assets 


Key  Points 

Protect  the  confidentiality  of  sensitive  data  while  distributing  and  archiving 
appraisal  assets.  Bundle  related  information  together  whenever  appropriate. 

Tools  and 
Techniques 

The  use  of  electronic  (database)  tools  for  managing  appraisal  data  often  provides 
assistance  in  ensuring  the  integrity  of  baselines,  as  well  as  repackaging 
information  for  archival  purposes.  Electronic  tools  allow  the  appraisal  team 
leader  to  remove  traceability  information  so  that  data  can  be  provided  to  the 
appropriate  people  while  preserving  the  anonymity  of  the  data  sources. 

Electronic  tools  also  support  the  submission  of  appraisal  data  to  the  SEI.  This  use 
of  electronic  tools  reduces  the  administrative  burden  and  facilitates  the  analysis 
of  appraisal  method  performance  data.  These  tools  also  provide  feedback  on  the 
consolidated  analysis  results  to  the  appraisal  community. 

Metrics 

While  archiving  and  reporting  the  metrics  associated  with  the  conduct  of  the 
appraisal  is  an  important  element  of  this  activity,  the  metrics  associated  with  the 
conduct  of  this  activity  itself  are  limited.  The  effort  and  calendar  time  consumed 
are  collected  and  compared  to  the  plan.  Some  appraisal  team  leaders  will  choose 
to  maintain  personal  metrics  associated  with  the  artifacts  described  in  this 
activity. 

Verification  and  The  Required  Practices  section  of  activity  3.2.2,  Generate  Appraisal  Record, 


Validation 

guides  the  verification  of  the  list  of  artifacts  provided  to  the  sponsor. 

The  Required  Practices  section  of  activity  3.2.3,  Provide  Appraisal  Feedback  to 
the  SEI  to  the  SEI,  guides  the  verification  of  the  list  of  artifacts  provided  to  the 
SEI.  Validation  is  provided  by  the  SEI  upon  receipt  of  the  appraisal  data 
package. 

Records 

•  appraisal  record 

•  appraisal  data  package 

•  lessons  learned 
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3.2  Package  and  Archive  Appraisal  Assets 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


As  the  final  process  in  the  appraisal,  this  process  is  about  collecting,  packaging, 
and  archiving  those  results  and  artifacts  produced  by  previous  processes  that 
become  part  of  the  appraisal  record.  Most  notably,  this  collection  of 
information  includes  the  appraisal  plan  and  appraisal  results.  Additionally, 
sensitive  or  proprietary  data  produced  by  other  appraisal  processes  must  be 
returned  to  the  organizational  unit  or  destroyed. 


This  process  performs  the  data  collection,  data  management,  and  reporting 
activities  necessary  to  close  out  the  appraisal.  Data  collected  throughout  the 
appraisal  is  consolidated  and  baselined,  becoming  a  permanent  part  of  the 
appraisal  record. 


♦ 
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3.2.1  Collect  Lessons  Learned 


Activity 

Description 

As  one  of  the  final  activities  in  wrapping  up  an  appraisal,  teams  typically  record 
lessons  learned  from  their  experience.  The  purpose  of  these  lessons  learned  is  to 
document  what  went  right,  what  went  wrong,  and  any  suggestions  or 
recommendations  for  improving  the  method  or  its  execution.  The  collection  of 
lessons  learned  is  a  recommended  activity  for  the  improvement  of  future 
appraisals,  but  is  not  a  method  requirement. 

Required 

Practices 

None 

Parameters 
and  Limits 

Lessons  learned  must  adhere  to  the  same  principles  of  confidentiality  and  non¬ 
attribution  applicable  to  other  appraisal  results. 

Implementation  Capturing  lessons  learned  is  often  done  as  a  group  at  the  end  of  the  appraisal, 


Guidance 

while  the  appraisal  activities  are  fresh  in  team  members’  minds.  This  collection 
of  lessons  learned  can  be  supplemented  with  additional  inputs  from  team 
members  upon  further  reflection,  if  necessary.  Appraisal  team  leaders  forward 
these  aggregate  lessons  learned,  as  appropriate,  to  various  stakeholders,  but 
always  to  the  other  team  members.  Appraisal  team  leaders  and  members  often 
maintain  summary  lists  of  appraisal  best  practices  and  lessons  learned  as  a 
mechanism  for  continuous  learning  and  improvement,  and  these  lists  are  used 
as  a  resource  for  planning  subsequent  appraisals. 

♦ 
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3.2.2  Generate  Appraisal  Record 


Activity 

Description 


Required 

Practices 


Appraisal  data  collected  throughout  the  appraisal  is  aggregated  and  summarized 
into  a  permanent  record  documenting  the  appraisal  conduct  and  results.  This 
collection  of  data  is  referred  to  as  the  appraisal  record  and  is  delivered  to  the 
appraisal  sponsor  for  retention. 


The  appraisal  team  leader  shall 

•  collect  and  baseline  appraisal  data  that  becomes  part  of  the  permanent 
records  provided  to  appraisal  stakeholders 

•  document  the  satisfaction  of  all  SCAMPI  A  requirements 

•  generate  the  appraisal  record  from  baselined  planning  and  execution  data 
collected  throughout  the  appraisal 

•  deliver  the  appraisal  record  to  the  appraisal  sponsor 
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3.2.2  Generate  Appraisal  Record 


Parameters 
and  Limits 


Required  contents  of  the  appraisal  record  include  the  following: 

•  appraisal  plan 

•  objective  evidence,  or  identification  thereof,  sufficient  to  substantiate  goal¬ 
rating  judgments 

•  characterizations  of  practice  implementation  determined  at  the  instantiation 
level  and  aggregated  at  the  organizational  unit  level 

•  identification  of  the  appraisal  method  (and  version)  used  along  with  any 
tailoring  options 

•  final  findings 

•  goal  level  ratings 

•  all  optional  ratings  rendered  during  the  appraisal  (Process  Area  ratings, 
capability  levels,  maturity  levels,  etc.) 

•  appraisal  disclosure  statement 

Appraisal  data  must  comply  with  rules  for  non-attribution,  confidentiality, 
protection  of  proprietary  information,  and  applicable  laws,  regulations,  or 
standards  (e.g.,  acquisition  regulations  or  security  classification).  Recipients  are 
expected  to  place  the  appropriate  limitations  on  the  access  and  use  of  the 
provided  appraisal  data. 

The  appraisal  team  leader  documents  in  the  appraisal  disclosure  statement  that  all 
SCAMPI  A  requirements  were  satisfied. 
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3.2.2  Generate  Appraisal  Record 


Implementation 

Guidance 


The  actual  objective  evidence  (artifacts  or  portions  of  artifacts)  need  not  be  part 
of  the  appraisal  record,  but  an  identification  of  the  objective  evidence  is 
required.  This  identification  may  be  implemented  by  providing  the  database  of 
objective  evidence  mapped  to  model  practices  that  were  used  as  the  basis  for 
characterizing  practice  implementation. 

Guidance  on  the  protection  of  appraisal  data  can  be  summarized  based  on  the 
recipient  of  the  data  as  follows: 

•  appraisal  sponsor:  Replacement  of  specific  sources  (persons,  basic  units, 
support  functions)  with  non-attributable,  general  identifiers  (e.g.,  numeric 
codes  assigned  to  basic  units,  support  functions,  roles,  or  data-gathering 
sessions).  If  the  sponsor  is  separate  from  the  appraised  organization  (e.g.,  in 
the  case  of  a  supplier  selection  context),  there  may  be  situations  where 
confidential  or  proprietary  data  relating  to  the  appraised  organization  must 
be  removed. 

•  SE1:  Same  as  for  appraisal  sponsor,  for  data  that  is  shared  by  both.  For  data 
that  is  provided  only  to  the  SE1,  the  data  collection  vehicles  (e.g.,  forms) 
are  already  designed  to  observe  non-attribution  and  confidentiality  rules. 
Additionally,  supplied  data  may  be  subject  to  further  sanitization  to  comply 
with  acquisition  or  security-related  restrictions. 

•  senior  site  manager:  In  cases  where  the  appraised  organizational  unit  is 
separate  from  the  appraisal  sponsor,  the  appraised  organization  is  typically 
provided  only  with  appraisal  results  and  not  data  related  to  planning  and 
decision  making,  or  data  that  makes  use  of  the  results. 


♦ 
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3.2.3  Provide  Appraisal  Feedback  to  the  SEI 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Implementation 

Guidance 


Appraisal  data  required  by  the  SEI  is  collected  and  reported.  This  data 
includes  a  subset  of  the  contents  of  the  appraisal  record,  as  well  other  data 
used  by  the  SEI  to  aggregate  and  analyze  appraisal  performance  data  for 
reporting  to  the  community  and  monitoring  the  quality  of  performed 
appraisals. 


The  appraisal  team  leader  shall 

•  submit  the  completed  appraisal  data  package  as  required  by  the  SEI 
The  appraisal  data  package  consists  of 

•  approved  appraisal  disclosure  statement 

•  approved  appraisal  plan 

•  the  final  findings  presentation  or  report 


The  SEI  defines  the  specific  set  of  data  required  for  submission  at  the 
completion  of  an  appraisal.  Submission  of  the  appraisal  data  package  is 
required  for  the  appraisal  to  be  recorded  in  the  SEI’s  database  of  appraisal 
results.  This  data  is  also  a  requirement  established  by  the  SEI  to  maintain 
SCAMPI  Lead  Appraiser  authorization. 


The  appraisal  team  leader  is  responsible  for  ensuring  that  the  appraisal  data 
package  is  collected  and  reported.  The  SEI,  as  custodian  of  the  product  suite 
and  the  Appraisal  Program,  has  several  objectives  in  seeking  appraisal 
feedback: 

•  characterization  of  the  state  of  the  practice  in  the  appraisal  community,  for 
the  collection  and  distribution  of  effective  appraisal  techniques 

•  analysis  of  reported  appraisal  data  to  obtain  an  understanding  of  appraisal 
performance  for  continuous  improvement 

•  quality  control  within  the  Appraisal  Program,  to  ensure  a  high  level  of 
confidence  in  the  accuracy  of  appraisal  results 

The  SEI  provides  approved  information  within  the  bounds  of  confidentiality  to 
the  community,  based  on  results  from  the  appraisal  data  collected.  The  SEI 
establishes  the  format  and  mechanisms  for  the  presentation  of  this  infonnation. 


♦ 
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3.2.4  Archive  and/or  Dispose  of  Key  Artifacts 


Activity 

Description 

After  the  various  reports  are  delivered  to  the  appropriate  stakeholders  and  the 
appraisal  assets  have  been  baselined,  the  appraisal  team  leader  is  responsible  for 
properly  archiving  and/or  disposing  of  the  appraisal  data,  in  accordance  with 
agreements  made  with  the  sponsor  and  documented  in  the  appraisal  plan.  The 
team  librarian  (if  one  is  used)  ensures  that  all  organization-provided 
documentation  and  objective  evidence  is  returned  or  disposed  of  properly.  Any 
remaining  team  artifacts  or  notes  are  disposed  of  properly. 

Required 

Practices 

The  appraisal  team  leader  shall 

•  archive  or  dispose  of  key  artifacts  collected  by  the  appraisal  team 

•  return  objective  evidence  provided  by  the  organizational  unit 

Parameters 
and  Limits 

In  all  usage  modes  of  SCAMPI  A,  strict  non-attribution  policies  apply. 
Confidentiality  and  non-disclosure  agreements  established  with  the  appraisal 
team  members  remain  in  effect. 
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3.2.4  Archive  and/or  Dispose  of  Key  Artifacts 


Implementation 

Guidance 


How  the  records  will  be  preserved  or  disposed  of  is  dependent  on  the  usage 
mode  of  the  method  and  the  appraisal  objectives  that  shape  the  current 
application.  Confidentiality  rules  may  differ  by  application.  In  a  supplier 
selection  usage,  the  results  are  not  proprietary  in  that  the  sponsor  is  not  a 
member  of  the  appraised  organization.  However,  results  are  only  known  to  the 
sponsor  and  the  recipient;  competing  organizations  do  not  see  the  results. 
Confidentiality  of  results  can  be  characterized  as  one  of  the  following: 

•  known  only  to  the  recipient  organization 

•  known  to  the  recipient  and  sponsor,  when  they  are  from  different 
organizations 

•  known  to  anyone 

The  sponsor  is  solely  responsible  for  determining  the  confidentiality  with  which 
the  appraisal  results  will  be  maintained.  The  non-attribution  of  data  to  specific 
individuals  is  the  responsibility  of  the  appraisal  team.  The  recipient 
organization,  if  the  sponsor  agrees  and  it  is  planned  for,  may  always  choose  to 
make  the  results  known  outside  the  organization.  At  a  high  level,  this  disclosure 
might  be  done  for  marketing  and  public  relations  reasons.  Disclosures  of  results 
include  the  context  and  constraints  under  which  the  appraisal  was  performed 
(e.g.,  appraisal  reference  model  scope  and  organizational  scope),  as  defined  by 
the  appraisal  disclosure  statement  described  in  process  3.1,  Deliver  Appraisal 
Results. 

Any  annotations  related  to  the  objective  evidence  provided  to  the  organization 
by  the  appraisal  team  should  be  recorded  and  archived  for  use  in  process 
improvement  actions  or  for  reuse  in  subsequent  appraisals. 


♦ 
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Appendix  A  The  Role  of  Objective  Evidence  in  Verifying 
Practice  Implementation 


Purpose 

This  appendix  provides  a  conceptual  overview  of  the  process  of  verifying  practice  implementation 
and  the  role  of  objective  evidence  in  that  process.  Verification  of  practice  implementation  is  an 
essential  element  of  appraising  the  implementation  of  processes  relative  to  models  of  best 
practices  such  as  the  CMMI  constellations  and  the  People  CMM. 

Verifying  Practice  Implementation 

In  this  discussion,  verifying  practice  implementation  means  the  substantiation  of  practice 
implementation  based  on  a  review  of  objective  evidence.  For  example,  one  might  inquire  as  to 
whether  a  practice  implemented  by  multiple  basic  units  is  substantiated  by  objective  evidence  for 
each  basic  unit.  Alternatively,  one  might  inquire  as  to  whether  an  organization-specific  practice  is 
implemented  within  an  organization. 

Having  a  well-defined  approach  for  verifying  practice  implementation  is  of  critical  importance 
from  several  perspectives.  For  the  process  improvement  sponsor,  it  provides  some  assurance  that 
the  resources  applied  to  the  improvement  effort  will  result  in  the  desired  outcome  and  that  the 
resultant  benefits  can  therefore  be  expected.  For  process  improvement  agents  or  champions,  it 
enables  them  to  know  when  they  have  succeeded  with  the  implementation  activity,  and  to 
informally  monitor  whether  the  practice  continues  to  be  implemented  over  time.  For  appraisal 
teams,  a  well-defined  verification  approach  is  essential  for  determining  what  capability  level  or 
maturity  level  ratings  are  warranted.  Goal  satisfaction  is  predicated  on  implementation  of  the 
relevant  practices  (or  acceptable  alternatives).  Hence  verification  of  practice  implementation  is  a 
crucial  appraisal  task. 

Determining  Practice  Implementation 

The  conduct  of  an  activity  or  the  implementation  of  a  practice  will  result  in  “footprints” — 
evidence  that  the  activity  was  conducted  or  the  practice  was  implemented. 

For  example,  if  one  balances  one’s  checkbook  at  the  end  of  the  month,  there  are  several  potential 
ways  to  confirm  that  this  activity  has  indeed  taken  place.  First,  the  person  who  engaged  in  the 
checkbook  balancing  activity  can  affirm  that  this  activity  was  conducted.  Second,  there  will  likely 
be  an  entry  in  the  checkbook  register  for  each  check  or  transaction  to  indicate  that  it  matches  with 
a  corresponding  entry  in  the  bank’s  statement.  Additional  artifacts  could  be  identified. 

The  general  idea  is  clear:  the  actual  conduct  of  an  activity  leaves  footprints  that  provide  a  basis  for 
verification. 


Objective  evidence  refers  to  the  footprints  that  are  the  necessary  and  unavoidable  consequence  of 
practice  implementation.  They  include  information  contained  in  artifacts  and  affirmation  gathered 
from  interviews  with  managers  and  practitioners. 
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The  Role  of  Objective  Evidence 

ARC-compliant  appraisal  methods  employ  one  or  more  types  of  objective  evidence1.  An  appraisal 
team  bases  its  decisions  about  practice  implementation  on  examination  of  this  objective  evidence. 

Once  a  basic  unit  or  support  function  has  an  understanding  of  how  its  processes  relate  to  the 
reference  model,  the  stage  is  set  for  capturing  the  objective  evidence  of  implementation.  The  work 
of  establishing  the  collection  of  objective  evidence  for  basic  units  and  the  support  functions 
mapped  to  model  practices  provides  assurance  to  the  process  improvement  sponsor  that  the 
expected  implementation  activities  have  in  fact  resulted  in  alignment  of  the  organization’s 
activities  with  the  reference  model. 

This  database  of  objective  evidence  is  itself  an  important  organizational  process  asset  that  has  a 
number  of  potential  uses,  most  notably  providing  an  appraisal  team  a  head  start  in  understanding 
the  organization’s  implementation  of  the  reference  model.  This  approach  leaves  the  appraisal 
team  the  task  of  verifying  whether  the  objective  evidence  provided  is  adequate  for  substantiation 
of  practice  implementation,  rather  than  the  more  difficult,  error  prone,  and  time-consuming  task 
of  investigating  each  practice  to  discover  the  objective  evidence  needed  to  substantiate 
implementation. 

Both  the  appraised  organization  and  the  appraisal  team  have  a  clearer  picture  of  what  artifacts 
must  be  provided  to  substantiate  implementation  of  the  practices,  thereby  minimizing  the  amount 
of  further  investigation  necessary  in  the  form  of  affirmations  and  additional  artifact  requests.  The 
extent  to  which  the  appraised  organization  can  provide  this  information  becomes  a  principal  factor 
in  how  much  further  investigation  may  be  required. 

Another  benefit  of  this  approach  is  significantly  greater  reliability  and  accuracy  of  appraisal. 

Populating  a  database  of  objective  evidence  mapped  to  model  practices  is  not  meant  to  turn  the 
appraisal  into  an  artifact  review  exercise.  It  merely  allows  for  more  focused  and  effective  use  of 
the  on-site  phase  and  potentially  a  shorter  on-site  phase  than  would  otherwise  be  the  case. 

Finally,  the  populated  database  is  not  intended  to  tie  the  hands  of  model  implementers  or  process 
appraisal  teams.  The  primary  value  of  the  populated  database  lies  in  making  explicit  what  has 
heretofore  been  implicit  and  therefore  subject  to  wide  variations  in  interpretation  and 
understanding.  Over  time,  sharing  of  populated  databases  will  result  in  a  set  of  practice 
implementation  scenarios  (e.g.,  small,  medium,  and  large  organizations,  work  groups  or  projects) 
and  a  standard  set  of  populated  databases  that  could  be  used  as  a  starting  point  for  further 
customization.  The  particular  process  implementation  context  and  the  specifics  of  the  instantiation 
would  detennine  which  objective  evidence  makes  sense  for  that  implementation.  Appraisal  teams 
would  be  obliged  to  inquire  into  the  existence  of  the  agreed-upon  objective  evidence,  while  still 
having  the  freedom  to  make  judgments  based  on  the  facts  and  circumstances  of  the 
implementation. 


The  ARC  defines  objective  evidence  as  “qualitative  or  quantitative  information,  records,  or  statements  of  fact 
pertaining  to  the  characteristics  of  an  item  or  service  or  to  the  existence  and  implementation  of  a  process  ele¬ 
ment,  which  are  based  on  observation,  measurement,  or  test  and  are  verifiable.” 
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A  standard  set  of  objective  evidence  could  establish  nonns  within  which  most  implementations 
will  fall,  thereby  allowing  efficiencies  to  be  realized  in  implementation  and  appraisal,  while  at  the 
same  time  recognizing  that  alternative  implementations  may  be  possible  using  alternative 
practices. 

Types  of  Objective  Evidence 

An  appraisal  team  bases  its  decisions  about  practice  implementation  on  the  existence  of  objective 
evidence  available  to  it.  This  objective  evidence  can  take  on  one  or  more  of  the  following  types: 

•  artifacts:  tangible  forms  of  objective  evidence  indicative  of  work  being  performed  that 
represents  either  the  primary  output  of  a  model  practice  or  a  consequence  of  implementing  a 
model  practice 

•  affirmations:  oral  or  written  statement  confirming  or  supporting  implementation  (or  lack  of 
implementation)  of  a  model  practice  provided  by  the  implementers  of  the  practice,  provided 
via  an  interactive  forum  in  which  the  appraisal  team  has  control  over  the  interaction, 
demonstrations  or  presentations  (e.g.,  the  demonstration  of  capability  of  a  tool  or  other 
mechanism  as  it  relates  to  the  implementation  of  a  practice,  or  a  presentation  explaining  some 
aspect  of  the  organization  or  one  or  more  of  its  basic  units) 


Objective  Evidence  Database 

An  objective  evidence  database  is  a  structure  or  schema  defined  to  provide  a  repository,  or  links 
to  a  repository,  for  the  objective  evidence  and  corresponding  mapping  to  model  practices.  Table 
12  shows  an  example  of  such  a  structure.  Note  that  this  example  is  a  notional  description  of  the 
content,  not  a  physical  definition  of  the  format. 


Table  12:  An  Objective  Evidence  Database  Schema 


Attribute 

Synopsis 

Remarks 

Practice  ID 

This  ID  identifies  the  process  area,  goal,  and  practice 
that  the  objective  evidence  is  associated  with. 

Acronyms  are  found  in  the 
reference  models. 

Objective 
Evidence  ID 

This  ID  identifies  the  type  of  objective  evidence. 

Types  are  artifact  and 
affirmation. 

Description 

This  attribute  is  a  description  of  the  objective  evidence 
as  applied  to  this  practice. 

Examples 

These  attributes  are  examples  of  artifacts  or 
affirmations  that  would  exemplify  the  intent  of  the 
objective  evidence  and/or  exploratory  questions  (EQs) 
or  “look  fors”  (LFs).  They  assist  appraisers  in 
identifying  relevant  artifacts  or  eliciting  relevant 
information. 

Aim  to  minimize  any 
overlap  with  such 
information  that  is  already 
in  the  model  document. 

Organizational 

Implementation 

This  attribute  would  be  filled  in  by  the  organization  as 
part  of  its  implementation  program  and  provided  to  the 
appraisal  team  as  a  resource. 
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Table  13  shows  an  example  objective  evidence  database  entry  for  specific  practice  1.1  for  CMMI- 
DEV  of  the  Project  Planning  process  area: 


Table  13:  Sample  Database  Record  Structure 


Attribute 

Value 

Practice  ID 

PP  SP  1.1 

Objective  Evidence  ID 

Artifact 

Objective  Evidence 
Description 

Work  product(s)  that  reflect  (document  the  information  content  of)  the 
establishment  of  a  top-level  work  breakdown  structure  (WBS)  to  estimate  of 
the  scope  of  the  project. 

Organizational 

Implementation 

To  be  provided  by  the  organization  for  a  specific  implementation 

These  descriptions  have  a  number  of  uses  in  addition  to  their  utility  during  process  appraisal. 

They  can  be  used  during  the  model  implementation  phase,  after  model  implementation  as  a 
training  vehicle  for  new  personnel,  for  internal  monitoring  of  practice  implementation,  etc. 

Application  of  Objective  Evidence  Databases  in  Model  Implementation 

The  use  of  objective  evidence  has  significant  utility  for  an  organization  that  is  committed  to 
model-based  process  improvement.  Typically,  organizations  will  either  implement  model 
practices  directly  or  will  ensure  that  the  practices  used  in  the  organization  affect  goal  achievement 
(through  the  mechanism  of  alternative  practices). 

Since  models  are  necessarily  expressed  in  an  implementation-independent  manner,  the 
implementation  of  a  model  will  require  that  an  understanding  of  how  the  model  intent  (as 
expressed  though  goals,  practices,  and  other  model  material)  is  to  be  realized  in  the  organization 
be  developed,  documented,  and  operationalized.  The  model  intent  is  made  real  through  its  impact 
on  the  way  people  work;  if  there  is  no  relation  between  how  they  work  and  the  model,  the 
organization  has  not  implemented  the  model.  Thus,  having  an  understanding  of  the  ways  in  which 
implementation  of  the  model  relates  to  what  people  are  doing  in  the  organization  is  a  necessary 
and  unavoidable  prerequisite  to  implementing  the  model.  Objective  evidence  databases  provide  a 
mechanism  by  which  the  implementation  of  a  model  practice  can  be  described. 

Application  of  Objective  Evidence  Databases  in  Process  Appraisal 

During  the  course  of  process  appraisal,  the  appraisal  team’s  primary  focus  is  on  verifying  practice 
implementation.  This  verification  is  accomplished  by  (1)  obtaining  objective  evidence  relevant  to 
the  implementation  of  a  practice,  (2)  comparing  the  objective  evidence  available  with  what  is 
expected,  and  then  (3)  making  a  determination  of  practice  implementation  based  on  the  difference 
between  actual  and  expected  evidence. 

The  database  assists  the  appraisal  team  (as  well  as  the  implementing  organization)  with  task  1  by 
providing  a  framework  or  structure  that  makes  explicit  the  types  of  objective  evidence  that  should 
be  considered.  In  concert  with  the  reference  model  documentation,  this  framework  provides  the 
model  basis  against  which  the  organization’s  actual  operating  practices  are  compared. 
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Note  that  the  database  does  not  prescribe  what  objective  evidence  must  be  present  for  practice 
implementation  determinations  to  be  made;  they  only  make  explicit  what  is  reasonable  for  an 
appraisal  team  to  consider.  The  particular  circumstances  and  attributes  of  the  instantiation  must  be 
taken  into  consideration  when  making  determinations  of  practice  implementation. 

The  database  structure  assists  the  appraisal  team  with  task  2  to  the  extent  that  the  team  has  agreed 
in  advance  on  the  objective  evidence  it  expects  to  see  for  each  process  instantiation  examined.  In 
some  cases  it  may  be  difficult  or  impossible  to  have  completely  developed  a  team  consensus  on 
what  objective  evidence  must  be  seen  (in  advance).  But  sooner  or  later  the  appraisal  team  must 
establish  a  consensus  view  on  what  is  reasonable  to  expect,  since  it  is  only  the  presence  of  that 
consensus  view  that  permits  a  determination  of  practice  implementation  to  be  made. 

The  final  practice  implementation  determination  task  is  that  of  developing  a  team  consensus  on 
whether  the  practice  is  implemented  for  the  process  instantiation  being  examined.  This  decision  is 
based  on  the  difference  between  what  is  expected  and  what  is  observed. 
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Appendix  B  Alternative  Practice  Identification  and 
Characterization  Guidance 


Concept  Description 

This  appendix  provides  guidance  on  how  to  identify  acceptable  alternatives  to  practices 
documented  in  the  reference  models  and  how  to  perform  practice  characterization  at  the 
instantiation  and  organizational  unit  levels  when  acceptable  alternative  practices  are  implemented 
in  lieu  of  model  practices. 

The  MDD  Glossary  includes  the  following  definition  of  “alternative  practice”: 

/)  practice  that  is  a  substitute  for  one  or  more  practices  contained  in  a  reference  model 
that  achieves  an  equivalent  effect  toward  satisfying  the  goal  associated  with  the  model 
practice.  Alternative  practices  are  not  necessarily  one-for-one  replacements  for  the 
model  practices. 

Analysis  and  use  of  alternative  practices  in  SCAMPI  A  appraisals  involves  the  following 
activities: 

•  identifying  which  model  practices  appear  to  be  implemented  using  an  alternative  practice,  and 
analyzing  whether  or  not  the  alternative  practice  does  indeed  achieve  an  effect  equivalent  to 
that  achieved  by  the  model  practices  toward  satisfying  the  associated  goal 

•  developing  an  instantiation-level  characterization  of  the  implementation  of  the  alternative 
practice  by  determining  whether  the  provided  evidence  includes  appropriate  artifact(s)  or 
affirmations  for  the  practice 

•  applying  the  instantiation-level  characterization  of  the  alternative  practice  to  the  model 
practice(s)  addressed 

•  aggregating  all  of  the  instantiation-level  characterizations  to  derive  the  organizational  unit- 
level  characterizations  and  generating  findings  and  goal  ratings,  just  as  is  performed  when  an 
alternative  practice  has  not  been  used 

Identification  of  Acceptable  Alternative  Practices 

Technically,  alternative  practices  can  be  discovered  at  any  time  during  an  appraisal,  up  to  and 
including  when  the  appraisal  team  is  analyzing  feedback  from  validation  of  the  preliminary 
findings.  However,  in  most  cases,  alternative  practices  are  uncovered  during  the  appraisal 
planning  and  preparation  activities.  Typically,  they  are  either  communicated  to  the  appraisal  team 
lead  by  the  organizational  unit  during  appraisal  planning  discussions  or  are  discovered  by  the 
appraisal  team  in  the  early  examinations  of  objective  evidence.  For  example,  little  or  no  evidence 
might  be  found  for  a  given  model  practice  or  set  of  related  model  practices,  but  evidence  for  an 
alternative  practice  might  be  uncovered  when  the  initial  data  set  is  analyzed  (activity  1 .4.2, 
Inventory  Objective  Evidence)  or  when  a  readiness  review  is  conducted  (activity  1.5.1,  Perform 
Readiness  Review). 
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The  alternative  practice  is  not  acceptable  until  the  appraisal  team  agrees  that  it  does  indeed 
achieve  an  effect  equivalent  to  one  or  more  model  practices.  To  do  so,  the  team  must  first  analyze 
the  alternative  practice  for  its  content  to  identify  which  model  practice(s)  it  addresses.  The 
appraisal  team  then  must  determine  what  effect  is  achieved  by  the  implementation  of  the 
addressed  model  practice(s)  toward  goal  satisfaction.  This  determination  is  more  than  simply  what 
work  product(s)  might  be  developed  as  a  result  of  implementation  of  the  model  practice(s). 

What  is  achieved,  supported,  and/or  enabled  as  a  result  of  implementation  of  the  practice(s)  and 
generation  of  the  work  product(s)?  What  information  becomes  available,  when,  and  to  whom? 
Once  these  questions  and  any  others  the  team  deems  appropriate  are  answered,  the  team  would 
need  to  decide  whether  the  alternative  practice  achieves  the  same  or  similar  results  or  just-as- 
effective  results  for  the  given  business  environment.  If  it  does,  then  it  achieves  an  equivalent 
effect  toward  satisfying  the  goal  associated  with  the  original  model  practice(s)  and  can  be 
considered  an  acceptable  alternative. 

As  noted  above,  an  alternative  practice  is  not  necessarily  a  one-for-one  replacement  for  a  given 
model  practice.  In  some  cases,  an  alternative  practice  might  only  partially  address  one  or  more 
model  practices.  In  such  cases,  the  alternative  practice  should  be  analyzed  in  conjunction  with  the 
associated  model  practice.  The  combined  implementation  could  be  acceptable.  In  other  cases,  an 
alternative  practice  might  consist  of  multiple  elements,  all  or  some  of  which  appear  to  address  one 
or  more  related  model  practices.  The  separate  elements  would  need  to  be  examined  to  ensure  that 
the  aggregate  of  those  elements  achieves  effect(s)  toward  goal  satisfaction  equivalent  to  the 
effect(s)  achieved  by  the  addressed  model  practice(s). 

Instantiation-Level  Characterization  of  Alternative  Practices 

Once  an  alternative  practice  has  been  approved  as  acceptable  by  the  appraisal  team,  its 
implementation  can  be  characterized  at  the  instantiation  level.  The  same  rules  are  used  for 
alternative  practices  as  those  applied  to  model  practices  (see  the  first  table  in  the  Parameters  and 
Limits  section  for  activity  2.4.2,  Characterize  Implementation  of  Model  Practices  and  Generate 
Preliminary  Findings).  However,  the  team  must  determine  what  artifacts  should  logically  be 
expected  from  implementation  of  the  alternative  practice.  Particular  attention  needs  to  be  paid  to 
whether  more  than  one  type  of  objective  evidence  might  be  needed  to  cover  the  complete 
implementation  approach.  This  situation  could  occur  especially  when  an  alternative  practice 

•  addresses  more  than  one  model  practice 

•  provides  only  partial  coverage  of  a  model  practice  and  is  combined  with  the  remaining 
aspects  of  that  model  practice  to  create  an  acceptable  alternative  practice 

•  consists  of  multiple  elements 

After  the  appraisal  team  decides  what  artifacts  it  should  expect  to  see,  the  team  can  examine  the 
evidence  provided  to  detennine  its  adequacy  and  coverage  of  the  alternative  practice  to 
characterize  the  implementation  of  that  practice. 
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Instantiation-Level  Characterization  of  the  Associated  Model  Practice(s) 

Typically,  the  characterization  of  the  alternative  practice  is  applied  directly  to  the  addressed 
model  practices.  This  characterization  is  straightforward  in  cases  where  the  alternative  practice 
addresses  a  single  model  practice.  However,  in  cases  where  an  alternative  practice  addresses  more 
than  one  model  practice,  the  characterizations  of  the  model  practices  may  vary  depending  on  the 
weaknesses  documented  for  the  alternative  practice  and  whether  the  evidence  provided  fully 
covers  the  alternative  practice  or  not.  The  weaknesses  associated  with  the  alternative  practice 
might  apply  to  only  some  of  the  addressed  model  practices.  For  example,  in  a  case  where  an 
alternative  practice  is  characterized  as  Largely  Implemented  or  Partially  Implemented,  some  of 
the  addressed  model  practices  could  still  be  characterized  higher  if  none  of  the  weaknesses  apply 
to  those  model  practice(s)  and  the  provided  evidence  is  adequate  for  those  practices. 

Organizational  Unit-Level  Characterization  and  Goal  Rating 

Once  the  addressed  model  practices  have  been  characterized  at  the  instantiation  level,  aggregation 
of  the  instantiation-level  practice  characterizations  proceeds  just  as  is  documented  in  the  second 
table  in  the  Parameters  and  Limits  for  activity  2.4.2,  Characterize  Implementation  of  Model 
Practices  and  Generate  Preliminary  Findings.  In  addition,  goal  rating  is  performed  just  as  is 
documented  in  activity  2.6.1,  Derive  Findings  and  Rate  Goals. 
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Appendix  C  Roles  and  Responsibilities 


Appraisal  Team  Lead  and  Members 

Appraisal  team  members  are  individuals  who  are  on  the  appraisal  team  during  the  onsite  period. 
“R”  indicates  the  responsibility  is  required.  “O”  indicates  the  responsibility  is  optional.  During 
appraisal  planning,  the  appraisal  team  leader  should  determine  which  optional  roles  and 
responsibilities  will  be  required  for  the  appraisal.  Mini-teams,  facilitator  and  time  keeper  roles  are 
not  specifically  required  by  name,  however,  the  responsibilities  associated  with  the  roles  (as 
indicated  by  R)  are  required  to  be  performed  by  the  appraisal  team. 


Table  14:  Appraisal  Team  Roles  and  Responsibilities 


Role 

Responsibility 

Required/ 

Optional 

Appraisal 

Team  Leader 

For  each  appraisal,  there  is  exactly  one  appraisal  team  leader.  The 
appraisal  team  leader  must  be  an  SEI-certified  SCAMPI  leader 
appraiser  for  each  reference  model  in  scope,  and  must  be  affiliated  to 
the  SEI  partner  that  is  responsible  for  the  appraisal. 

R 

Has  overall  responsibility  for  the  appraisal 

R 

Meets  with  the  sponsor  prior  to  the  appraisal  to  discuss  appraisal 
scope  and  other  appraisal  planning  parameters 

R 

With  the  support  of  the  appraisal  coordinator,  determines  process 
areas  that  are  not  applicable  or  out  of  scope 

R 

With  the  support  of  the  appraisal  coordinator,  selects  basic  units  from 
sub-groups 

R 

With  the  support  of  the  appraisal  coordinator,  creates  and  completes 
the  appraisal  plan  and  detailed  schedule 

R 

Signs  the  appraisal  plan 

R 

Meets  with  the  sponsor  to  discuss  outcomes  of  the  readiness  review 
and  jointly  decides  whether  the  appraisal  should  proceed  as  planned, 
be  re-scheduled,  or  be  cancelled 

R 

Provides  appraisal  participants  an  overview  of  the  appraisal  process 
and  schedule 

R 

Assigns  team  roles 

R 

May  be  a  member  of  a  mini-team;  however,  due  to  additional 
responsibilities  as  the  appraisal  team  leader,  mini-team  responsibilities 
may  be  reduced  or  delegated  to  other  appraisal  team  members 

0 

Facilitates  team  resolution  of  conflicts  and  impasses 

R 

Monitors  schedule  and  performance 

R 

Votes  on  appraisal  outcomes 

R 

Ensures  the  SCAMPI  appraisal  process  is  followed 

R 
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Role 

Responsibility 

Required/ 

Optional 

Delivers  the  preliminary  findings 

R 

Ensures  the  final  findings  are  delivered 

R 

Signs  the  final  findings  and  the  appraisal  disclosure  statement 

R 

Reports  results  to  the  SEI 

R 

Mini-Teams  or 
appraisal  team 
members 

The  appraisal  team  members  are  typically  assigned  to  mini-teams  in 
groups  of  two  to  three  members  each.  Mini-teams  are  typically 
organized  by  related  process  areas  (e.g.,  process  area  categories), 
organizational  entities  or  by  discipline  and  appraisal  experience. 

0  (use  of 
mini-teams 
is  optional) 

Reviews  evidence  for  their  assigned  process  area  or  basic 
unit/support  function 

R 

Requests  additional  evidence  needed  relative  to  their  process  areas  or 
basic  units/support  function 

R 

Records  review  results  pertaining  to  their  process  areas  or  basic 
units/support  functions 

R 

Consolidates  findings  prior  to  full  team  consolidation 

R 

Votes  on  appraisal  outcomes 

R 

Signs  the  final  findings 

R 

Facilitator 

This  role  is  typically  assigned  to  the  appraisal  team  lead  or  an 
appraisal  team  member  for  each  affirmation  session.  The  individual 
may  differ  between  affirmation  sessions. 

R 

Ensures  interviewees  are  aware  of  confidentiality  and  non-attribution 
rules 

R 

Conducts  affirmations 

R 

Ensures  appraisal  team  members  take  notes  during  the  affirmations 

R 

Timekeeper 

This  role  is  typically  assigned  to  an  appraisal  team  member. 

0 

Tracks  time  and  schedule  constraints  during  affirmations  and  other 
activities 

0 

Other  Appraisal  Participants 

Other  appraisal  participants  are  individuals  who  may  support  the  appraisal  prior  to,  during,  and 
after  the  onsite  period;  however,  they  are  not  on  the  appraisal  team.  “R”  indicates  the 
responsibility  is  required.  “O”  indicates  the  responsibility  is  optional.  During  appraisal  planning, 
the  appraisal  team  leader  and  “appraisal  coordinator”  should  determine  which  optional 
responsibilities  will  be  required  for  the  appraisal.  The  responsibilities  of  the  Appraisal 
Coordinator  may  be  fulfilled  by  more  than  one  individual  and  may  be  referred  to  by  other  titles 
(e.g.,  site  coordinator,  librarian). 
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Table  15:  Other  Appraisal  Participants  -  Roles  and  Responsibilities 


Role 

Responsibility 

Required/ 

Optional 

Appraisal 

Coordinator 

This  role  is  typically  performed  by  an  individual  or  group  within  the 
organizational  unit  who  supports  planning  of  the  appraisal  and  prepares  basic 
units/support  functions  for  the  appraisal.  However,  the  appraisal  team  leader 
may  perform  any  of  these  responsibilities. 

R 

1. 

Defines  activities  and  schedules  to  prepare  basic  units/support  functions 
for  the  appraisal 

0 

2. 

Supports  the  appraisal  team  leader  to  determine  process  areas  that  are 
not  applicable  or  out  of  scope 

0 

3. 

Works  with  the  Appraisal  Lead  to  determine  appraisal  dates 

0 

4. 

Assists  in  selection  of  any  internal  appraisal  team  members 

0 

5. 

Supports  the  appraisal  team  leader  to  select  basic  units  and  support 
functions 

0 

6. 

Assists  the  appraisal  team  leader  in  completing  the  appraisal  plan  and 
detailed  schedule 

R 

7. 

Assists  the  basic  units/support  functions  in  selecting  appraisal  participants 

0 

8. 

Ensures  interviewees  arrive  to  affirmations  on  time 

0 

9. 

Assists  basic  units/support  functions  to  address  appraisal  findings  after  the 
appraisal 

0 

10. 

Uses  the  strengths  and  weaknesses  from  appraisals  to  improve  processes 

0 

Appraisal 

Sponsor 

This  role  is  assigned  to  the  individual  who  sponsors  the  appraisal  and  approves 

the  appraisal  plan. 

R 

1. 

Defines  the  initial  organizational  unit  to  be  appraised 

R 

2. 

Explains  the  value  of  process  improvements  to  upper  management 

0 

3. 

Ensures  budget  is  provided  to  support  the  appraisal 

R 

4. 

Meets  with  the  appraisal  team  leader  prior  to  the  appraisal  to  discuss 
appraisal  scope  and  other  appraisal  planning  parameters 

R 

5. 

Selects  the  appraisal  team  leader 

R 

6. 

Selects  the  appraisal  coordinator 

0 

7. 

Reviews  and  approves  the  appraisal  plan 

R 

8. 

Meets  with  the  appraisal  team  leader  to  discuss  outcomes  of  the  readiness 
review  and  jointly  decides  whether  the  appraisal  should  proceed  as 
planned,  be  re-scheduled,  or  be  cancelled 

R 

9. 

Attends  an  executive  briefing 

0 

10. 

Attends  the  final  findings  presentation 

R 
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Role 

Responsibility 

Required/ 

Optional 

1 1 .  Signs  the  appraisal  disclosure  statement 

R 

Appraisal 

Participant 

The  appraisal  participant  role  applies  to  members  of  basic  units  or  support 

functions  who  provide  artifacts  or  participate  in  affirmation  activities  such  as 

interviews,  or  demonstrations. 

R 

1 .  Attends  the  participant  briefing 

R 

2.  Works  with  the  appraisal  coordinator  or  appraisal  team  leader  to  provide 
artifacts  in  support  of  the  appraisal  prior  to  and  during  the  conduct 
appraisal  phase 

R 

3.  Attends  affirmation  sessions  as  initially  scheduled  and  as  requested  for 
follow-up 

R 

4.  At  least  one  individual  from  each  support  function  and  basic  unit 

participates  in  the  validation  of  preliminary  findings  and  provides  additional 
evidence  to  resolve  preliminary  finding  issues 

R 

5.  Attends  the  final  findings  presentation 

0 
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Role  Obligations  and  Access  Rights 

“R”  indicates  the  role  is  required  to  participate  in  the  activity.  “O”  indicates  the  role  may 
participate  in  the  activity  at  their  discretion,  or  delegate  the  activity  to  another  individual.  “N” 
indicates  the  role  is  not  allowed  to  participate  in  the  activity  (no  access)  unless  requested  by  the 
appraisal  team  leader. 

Table  16:  Appraisal  Team  -  Role  Obligations  and  Access  Rights 


Team  Training 

Perform  Readiness  Review 

Readiness  Decision 

Participant  Briefing 

Evidence  Review  Activities 

Mini-Team  Consolidation  (if  used) 

Affirmations 

Team  Consensus 

Preliminary  Findings  Validation 

Final  Findings  Delivery 

Appraisal  Team  Leader 

R 

R 

R 

R 

R 

01 

01 

R 

R 

R 

Appraisal  Team  Member 

R 

0 

0 

R 

R 

R 

R2 

R 

R 

0 

Appraisal  Coordinator 

N 

0 

0 

0 

N 

N 

N 

N 

N 

0 

Sponsor 

N 

0 

R 

0 

N 

N 

N 

N 

N 

R 

Appraisal  Participant 

N 

0 

0 

R3 

N 

N 

R3 

N 

R3 

0 

1  The  appraisal  team  leader  has  additional  responsibilities  and  may  or  may  not  be  able  to  support 
mini-team  team  consolidation  and  all  affirmations. 

2  At  least  two  team  members  are  required  to  attend  their  affirmations.  Team  members  are  not 
required  to  attend  all  affirmations  unless  requested  by  the  appraisal  team  leader. 

3  Applies  to  at  least  one  person  from  each  support  function  and  basic  unit. 


CMU/SEI-201 1-HB-001  |  199 


CMU/SEI-201 1-HB-001  |  200 


Appendix  D  Reporting  Requirements  and  Options 


The  reporting  requirements  for  SCAMPI  appraisals  are  documented  throughout  Part  II  of  the 
Method  Definition  Document.  The  following  table  provides  lead  appraisers  with  a  concise  picture 
of  what  needs  to  be  submitted,  when  it  needs  to  be  submitted,  to  whom  it  needs  to  be  submitted, 
and  the  relevant  sections  in  the  MDD  where  its  content  and  submittal  is  discussed.  In  some  cases, 
items  are  required  to  be  submitted  in  accordance  with  policy  set  by  the  SEI  rather  than  required 
practices  in  the  MDD. 


Table  1 7:  Submissions  Requirements  for  SCAMPI  Appraisals 


Item 

When  Due 

Recipient(s) 

Relevant  MDD  Section 

Appraisal 

plan 

May  be  generated  incrementally  throughout 
planning,  but  must  be  approved  prior  to  the 
start  of  Conduct  Appraisal  phase.  Portions 
must  be  submitted  with  the  initial  SAS  record 
for  the  appraisal  30  days  prior  to  the  Conduct 
Appraisal  phase. 

SEI 

1 .2.7  Obtain  Commitment  to 
Appraisal  Plan 

Final  findings 

Presented  or  provided  to  the  sponsor  prior  to 
the  conclusion  of  the  Conduct  Appraisal 
phase 

Sponsor 

SEI 

2.6.5  Document  Appraisal 

Results 

3.1  Deliver  Appraisal  Results 

3.1.1  Deliver  Final  Findings 

Appraisal 

Disclosure 

Statement 

A  draft  is  provided  to  the  sponsor  with  the 
final  findings.  The  final  appraisal  disclosure 
statement  is  created  with  the  closure  of  the 
SAS  record  for  the  appraisal  and  is  included 
with  the  appraisal  record  and  the  package  of 
appraisal  data  provided  to  the  SEI  within  30 
days  of  the  conclusion  of  the  Conduct 
Appraisal  phase. 

Sponsor 

SEI 

3.1.1  Deliver  Final  Findings 

3.1.2  Conduct  Executive 
Session(s) 

3.2.2  Generate  Appraisal 

Record 

3.2.3  Provide  Appraisal 

Feedback  to  the  SEI 

Appraisal 

Record 

Prior  to  the  completion  of  the  Report  Results 
phase 

Sponsor 

3.2  Package  and  Archive 
Appraisal  Assets 

3.2.2  Generate  Appraisal 

Record 

Package  of 
Appraisal 

Data  for  the 

SEI 

Within  30  days  of  the  conclusion  of  the 
Conduct  Appraisal  phase 

SEI 

3.2.3  Provide  Appraisal 

Feedback  to  the  SEI 
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Appendix  E  Managed  Discovery 


“Managed  Discovery”  is  a  phased  data  collection  approach  beginning  with  an  initial  data  call  for  a 
pre-determined  set  of  artifacts,  followed  by  a  set  of  iterative  calls  based  on  the  appraisal  team’s 
evaluation  of  the  work  products  and  remaining  evidence  gaps.  This  approach  can  be  employed  to 
efficiently  use  the  resources  required  to  prepare  data  in  support  of  SCAMPI  and  minimize  the 
overall  data  collection  effort.  It  represents  a  “middle  ground”  between  the  legacy  “discovery”  and 
“verification”  approaches  that  have  been  part  of  SCAMPI  since  its  inception. 

Background 

Many  organizations  invest  a  significant  amount  of  time,  effort  and  funds  in  preparing  the  data 
required  to  conduct  a  SCAMPI  appraisal.  In  fact,  it  appears  that  some  organizations  spend  more 
resources  in  preparing  the  data  for  SCAMPI  than  in  supporting  all  of  the  other  appraisal  activities. 
This  is  often  true  for  organizations  that  are  performing  SCAMPI  for  the  first  time,  or  are  trying  to 
achieve  higher  levels  of  maturity  or  capability.  Over-emphasis  on  the  idea  of  “verification”  has,  in 
many  cases,  led  to  the  belief  that  all  appraisal  data  must  be  identified  and  prepared  by  the 
organization  in  advance  of  the  appraisal  event  itself  —  rather  than  relying  on  the  appraisal  team  to 
be  diligent  in  seeking  out  any  additional  information  needed  to  support  the  appraisal.  The  concept 
of  “discovery”  by  the  appraisal  team  has  become  synonymous  with  “risk  of  failure”  to  many 
senior  managers  in  appraised  organizations.  As  a  consequence,  many  organizations  over-prepare 
for  a  “verification-based”  appraisal.  With  this  verification  approach,  artifacts  may  be  supplied  that 
are  not  applicable  and/or  are  never  reviewed  by  the  appraisal  team.  This  can  generally  be 
attributed  to  the  organization  misinterpreting  the  appraisal  reference  model  and/or  the  perceived 
appraisal  team’s  need  for  artifacts.  The  overall  result  is  typically  a  significant  expense  in  terms  of 
time,  effort  and  funds  to  support  the  data  collection  activities  of  the  appraisal. 

Summary  of  Approach 

Managed  Discovery  attempts  to  balance  verification  and  discovery  activities  by  using  an 
interactive,  phased  approach  to  data  collection  to  mitigate  the  risk  of  the  organization  failing  to 
provide  the  needed  data  or  providing  inappropriate  data.  The  key  is  starting  with  an  initial  set  of 
artifacts  that  have  the  most  general  reference  model  applicability.  This  allows  the  organization  to 
efficiently  provide  data  that  have  the  greatest  likelihood  of  being  useful  to  the  appraisal  team.  The 
appraisal  team  reviews  the  provided  artifacts  and  maps  them  to  the  appropriate  appraisal  reference 
model  practices.  The  appraisal  team  then  enters  into  an  iterative  set  of  data  calls  which  continue 
until  the  full  set  of  relevant  data  are  examined  (per  the  appraisal  scope).  This  approach  shifts  the 
responsibility  to  focus  the  data  collection  and  mapping  activity  to  the  appraisal  team  -  rather  than 
potentially  leaving  the  organization  guessing  at  what  the  team  needs.  The  initial  series  of  data 
calls  may  occur  during  initial  planning  activities,  during  readiness  reviews,  or  during  class  C  and 
class  B  appraisals  that  lead  up  to  the  class  A  event. 
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The  key  to  successful  implementation  of  this  approach  is  interaction  between  the  appraisal  team 
and  the  organization.  The  team  works  with  experts  from  the  organization  who  understand  the 
contents  of  the  products  being  provided  to  ensure  coverage  of  the  practices  in  the  model.  This 
leads  to  a  greater  understanding  of  the  organization  and  its  work  by  appraisal  team  members,  and 
can  serve  to  further  highlight  the  connection  between  “doing  the  work”  and  the  practices  in  the 
model  for  the  organization’s  staff. 

Defining  and  Reviewing  the  Initial  Data  Call 

The  initial  data  call  is  organized  around  key  products  with  high  model  leverage  (e.g.,  one-to-many 
relationships  across  practices,  such  as  plans,  schedules,  financial  reports,  and  reviews).  A  list  of 
example  “high  yield”  work  products  is  provided  at  the  end  of  this  appendix.  This  product-centric 
approach  allows  the  organization  to  align  the  data  more  closely  with  the  way  they  actually  do 
business.  For  example,  any  CMMI-DEV  appraisal  that  includes  the  Project  Planning  process  area 
would  likely  find  value  in  reviewing  the  “Project  Plan.”  Based  upon  past  experiences,  advanced 
organizations  can  scale-up  this  initial  data  increment  to  gain  greater  efficiency.  For  the  less 
experienced  organizations,  the  choice  to  limit  the  initial  increment  is  made  in  order  to  reduce  the 
risk  of  spending  effort  to  gather  artifacts  that  provide  little  or  no  value  to  the  appraisal  process. 

The  appraisal  team  leader  would  be  responsible  for  providing  the  target  contents  of  this  initial 
data  call  to  the  organization.  The  organization  focuses  on  providing  the  data  requested,  centering 
on  the  work  performed,  and  not  on  the  model  practices.  The  appraisal  team,  in  whole  or  in  part,  is 
responsible  for  mapping  the  contents  of  this  data  to  the  model  practices.  The  appraisal  team, 
working  with  experts  from  within  the  organization,  reviews  their  mapping  of  the  data  to  model 
practices  and  identifies  areas  where  evidence  is  missing,  or  where  additional  support  might  be 
needed.  These  results  are  documented  and  used  to  prepare  the  next  data  call. 

In  this  approach  to  data  collection,  the  initial  call  and  at  least  one  initial  review  should  be  done 
prior  to  the  beginning  of  the  Conduct  Appraisal  phase,  before  the  “90  day  clock”  starts. 


Specifying  the  Subsequent  Data  Calls 

Based  on  the  results  of  processing  the  initial  data  set,  another,  more  focused  data  call  can  be 
specified.  This  allows  the  team  to  refine  the  wording  used  to  describe  what  is  needed,  based  on 
what  was  learned  from  reviewing  the  initial  data  set.  As  well,  the  team  can  sharpen  their  focus  on 
topics  that  appear  to  be  potentially  problematic  or  implemented  in  a  unique  way  in  the 
organization.  Prioritizing  these  areas  to  be  investigated  early  will  provide  the  organization  with 
more  opportunities  to  supply  the  right  information.  It  will  also  allow  the  appraisal  team  members 
to  identify  areas  requiring  follow-up  through  interviews  or  demonstrations,  thus  further  refining 
the  data  collection  plan. 

Earlier  calls  will  tend  to  focus  primarily  on  data  that  might  be  missing  from  the  evidence 
database.  Later  calls  may  focus  more  on  the  appropriateness  of  the  information.  These  data  calls 
can  be  repeated  until  the  appraisal  team  feels  the  data  provides  sufficient  coverage  of  the 
reference  model  scope  and  that  the  data  is  sufficient  for  them  to  be  able  to  characterize  the 
practices.  These  review  sessions  might  coincide  with  readiness  reviews,  and  possibly  early 
verification  of  the  artifacts,  which  would  begin  the  Conduct  Appraisal  phase  and  the  90  day  clock. 
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Risk  of  Employing  Managed  Discovery 

It  is  important  to  remember  that  the  objective  of  data  collection  for  the  appraisal  is  to  ensure  that 
the  data  provided  by  the  organization  adequately  covers  the  reference  model  practices  in  scope. 
Therefore,  it  is  important  that  the  objective  evidence  database  clearly  map  to  the  appropriate 
practices.  The  use  of  product-centered  data  collection  without  prior  development  of  a  mapping  to 
the  reference  model  (i.e.,  Managed  Discovery)  can  increase  the  risk  of  missing  areas  of  concern  to 
the  organization  from  the  perspective  of  model  compliance.  Effective  use  of  interviews  by  the 
appraisal  team  can  mitigate  this  risk. 

There  will  also  be  organizations  for  which  maturity  level  ratings  risk  is  the  primary  concern.  Such 
organizations  may  find  that  creating  their  own  model-based  data  structures  across  the  sampled 
basic  units  and  support  functions  best  mitigates  this  risk,  despite  the  resources  and  costs  involved. 

Summary 

Managed  Discovery  represents  a  third  data  collection  approach  in  addition  to  discovery  and 
verification.  It  allows  the  organization  to  efficiently  provide  key  artifacts  that  will  demonstrate 
implementation  of  multiple  reference  model  practices.  It  allows  the  appraisal  team  to  identify 
specific  data  needed  to  enable  them  to  make  judgments  on  that  implementation.  This  is  in  contrast 
to  “verification  only”  appraisals  where  the  organization  provides  “everything”  and  the  team  must 
try  to  pick  out,  from  all  the  data  provided,  the  “few  critical  things”  that  require  follow-up  actions. 
It  can  help  minimize  the  negative  perception  of  “discovery”  in  appraisals,  by  providing  an 
interactive  build-up  of  the  organization’s  objective  evidence  database.  The  concepts  of  continuous 
consolidation  and  triage  of  appraisal  data  have  been  a  part  of  SCAMPI  from  its  earliest  versions. 
The  use  of  a  Managed  Discovery  approach  can  support  the  organization  more  completely. 
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Example  High-Yield  Work  Products 

The  following  table  is  intended  to  provide  some  examples  of  key  work  products  that  can  provide  one-to-many  relationships  to  multiple  practices  in  the 
reference  model  in  scope.  These  products  could  be  considered  as  part  of  the  initial  data  call  for  use  by  the  appraisal  team. 

This  list  is  not  intended  to  be  exhaustive,  nor  does  it  represent  any  requirement  for  specific  products  the  organization  and  its  sampled  units  must 
provide. 


Table  18:  Examples  of  Key  Work  Products 


Category 

CMM-DEV 

CMMI-SVC 

CMMI-ACQ 

People  CMM 

Planning  Products 

•  Program  Management  Plan  (PMP) 

•  Integrated  Master  Plan  and  Integrated 
Master  Schedules  (IMP,  IMS) 

•  Configuration  Management  Plan  (CMP) 

•  Systems  Engineering  Management  Plan 
(SEMP) 

•  Software  Development  Plan  (SDP) 

•  Quality  Assurance  Plans 

•  Training  Plans 

•  Measurement  Plans 

•  Estimating  records 

•  Release  planning 

•  Workflow  planning 

•  Kanban  boards 

•  Service  Catalogue 

•  Service  strategy 

•  Work  Plan 

•  Staffing  and  Capacity  Plans 

•  Quality  Assurance  Plans 

•  Training  Plans 

•  Measurement  Plans 

•  Estimating  records 

•  Orders 

•  Proposals 

•  Architectural/  building  /interior 
layout  diagrams 

•  Customer  instructions 

•  Posted  policies  and  flow- 
diagrams 

•  Work  flow  planning 

•  Kanban  boards 

•  Acquisition  Strategy 
Documents 

•  Supplier  Evaluation 
Criteria 

•  Requests  for  Proposal 

•  Specific  acquisition  plans 

•  Evaluation  plans 

•  Workgroup  Plan 

•  Workgroup  Training  Plan 

•  Individual  Training  and 
Development  Plan 

•  Organization  schedule  for 
performance  management 
activities 

•  Business  Plan 

•  Staffing  Plan 

•  Work  Environment  Plans 
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Category 

CMM-DEV 

CMMI-SVC 

CMMI-ACQ 

People  CMM 

Financial 

Management 

Products 

•  Budget  records 

•  Earned  Value  reports 

•  Cycle  time 

•  Takttime 

•  Lead  time 

•  Other  financial  reports 

•  Budget  vs.  actual  records 

•  Timesheets 

•  Personnel  schedules 

•  Inventory/  Supply  management 

•  Invoices 

•  Accounts  payable/accounts 
receivable 

•  Cycle  time 

•  Takttime 

•  Lead  time 

•  Other  financial  reports 

•  Budget  records 

•  Earned  Value  reports 

•  Supplier  invoices 

•  Operations  and  support 
costs 

•  Other  financial  reports 

•  Workgroup  budget  and 
actual  records 

•  Compensation  and 
Compensation  Adjustment 
Reports 

Management 

Products 

•  Regular  program  status  reports  (i.e.,  daily/ 
weekly/  monthly/quarterly  reviews) 

•  Records  of  major  project  milestones  (e.g., 
Preliminary  Design  Reviews,  deliveries) 

•  QA  Audit  records/reports 

•  Measurement  reports/repository 

•  Kanban  board 

•  Continuous/Cumulative  Flow  diagrams  and 
analysis 

•  Service  Agreements/SLAs 

•  Performance  Reports,  such  as 
capacity  and  availability  tools, 
customer  praise/  complaints, 

•  QA  Audit  records/reports 

•  Measurement  reports/repository 

•  Timesheets 

•  Personnel  schedules 

•  Kanban  board 

•  Continuous/  Cumulative  Flow 
diagrams  and  analysis 

•  Supplier  agreements 

•  QA  Audit  records/reports 

•  Measurement 
reports/repository 

•  Process  Verification 

Reports 

•  Process  Measurements 

•  Individual  Performance 
Management  records 

•  Workgroup  Goals  and 
Measures  of  Success 

•  Individual  Goals  and 
Measures  of  Success 
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Category 

CMM-DEV 

CMMI-SVC 

CMMI-ACQ 

People  CMM 

Execution 

Products 

•  Requirements  Documents  or  reports  from 
requirements  tools 

•  Concept  of  Operations  documents 
(CONOPS) 

•  Interface  Control  Documents  or  equivalent 

•  Design  Documents 

•  Verification  Plans 

•  Validation  Plans 

•  Integration  Plans  and  Procedures 

•  Test  Plans,  Procedures,  Reports 

•  Change  Control  Board  records 

•  Completed  tasks  /  products/components 

•  Tasks  being  tracked 

•  Burn-down/up  charts 

•  Kanban  board 

•  Continuous/  Cumulative  Flow  Diagram 

•  Peer  review  results 

•  These  are  likely  to  be  quite 
specific  to  the  service  type. 

That  is,  training  records,  for 
training  service,  diagnostic 
metrics  in  health  care, 
completed  orders,  served  food, 
order/service/  request  tracking 
systems,  Kanban  board,  etc. 

•  Customer  and 
contractual  requirements 

•  Supplier  design 
documents 

•  Completed  supplier 
evaluations 

•  Supplier  status 
reports/meeting  minutes 

•  Product/Service 
validation  reports 

•  QA  audit  reports/records 

•  Acceptance  documents 
for  delivered  products  or 
services 

•  Verification  Plans 

•  Validation  Plans 

•  Sales  Reports 

•  Records  of  performance 
discussions  with  Manager 

•  Training  Records 

•  Status  of  Performance 
Management  Activities 

•  Status  of  New  Hire 
requisitions 

•  Communication  and 
Coordination  Reports 

Tools/Repositories 

•  Requirements  tools 

•  Configuration  Management  tools 

•  Kanban  board 

•  Wikis 

•  Incident  tracking  tools 

•  Service  Request  processing 
tools 

•  Kanban  board 

•  Wikis 

•  Supplier  databases 
(preferred  supplier  tools, 
etc.) 

•  Metrics  Tools 

•  Performance  Management 
Tools/Systems 

•  Skills  Databases 

•  Education  and  Training 
tracking  tools 
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Appendix  F  Scoping  and  Sampling  in  SCAMPI  A  Appraisals 


Introduction 

The  sampling  process  for  appraisals  must  provide  clear  operational  definitions,  be  robust  to  a 
variety  of  different  applications,  and  promote  confidence  in  appraisal  results.  The  expectation  that 
appraisals  are  based  on  a  representative  sample  requires  a  clear  definition  for  what  is  meant  by 
“representative.”  Use  of  the  appraisal  method  with  organizations  of  various  sizes  and  types 
requires  the  terminology  and  definitions  have  broad  applicability  to  these  contexts.  Confidence  in 
the  results  of  appraisals  is  enhanced  through  visibility  of  the  breadth  and  depth  of  involvement 
from  different  parts  of  the  organization.  These  goals  are  addressed  in  the  process  described  here. 

The  process  of  defining  a  representative  sample  relies  on  the  skilled  analysis  of  the  way  people 
are  organized,  the  contextual  factors  that  can  lead  to  diversity  in  the  way  work  is  performed,  and 
use  of  a  standard  method  for  establishing  a  balanced  set  of  examples  for  the  appraisal  team  to 
examine.  Requirements  for  prerequisite  experience  and  training  confirm  the  capability  of  lead 
appraisers  to  apply  the  SCAMPI  method  in  the  settings  appropriate  to  their  certifications. 

Required  analysis  of  contextual  factors  is  guided  by  the  definition  of  Sampling  Factors  in  the 
method,  which  is  reinforced  through  the  required  training  provided  to  lead  appraisers.  The  process 
of  establishing  a  representative  (or  balanced)  sample  relies  on  an  objectively  verifiable 
computation,  which  is  also  defined  in  the  method  and  addressed  in  required  training. 

Successful  use  of  the  sampling  process  relies  on  adequate  understanding  of  core  concepts  by  all 
parties  who  have  a  stake  in  the  appraisal  process.  The  lead  appraiser  is  responsible  for  assuring 
that  the  appraisal  sponsor  has  this  understanding,  and  operating  within  the  requirements  of  the 
method. 

Defining  the  Organizational  Unit 

The  concept  of  an  “organizational  unit”  defines  boundaries  within  the  organization  to  which  the 
appraisal  results  apply.  The  lead  appraiser  works  with  the  appraisal  sponsor  to  establish  an 
appropriate  specification  of  this  boundary.  The  appraisal  sponsor  ensures  the  scope  defined 
represents  the  needs  of  the  business  for  establishing  a  credible  benchmark.  The  lead  appraiser 
ensures  the  requirements  of  the  SCAMPI  process  are  satisfied  to  support  accurate  and  credible 
statements  of  appraisal  results. 

The  sponsor  typically  has  an  initial  expectation  about  the  scope  of  the  appraisal.  If  the 
organization  has  participated  in  appraisals  in  the  past,  the  sponsor  may  request  a  re-appraisal  of 
the  same  organizational  unit.  While  the  work  underway  (products  and  services  being  delivered  at 
that  time)  may  differ  from  the  previous  appraisal,  the  definition  of  the  organizational  unit  used  in 
the  previous  appraisal  is  a  good  starting  point  for  the  sampling  process. 

The  process  of  defining  the  organizational  unit  (OU)  is  often  carried  out  in  an  iterative  fashion,  in 
which  tradeoffs  between  breadth  of  scope  and  data  collection  costs  drive  key  decisions.  The  lower 
boundary  for  the  size  of  an  organizational  unit  is  determined  by  the  model  scope  of  the  appraisal, 
because  the  model  scope  determines  the  minimum  set  of  examples  of  work  that  must  be  included 
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(the  minimum  model  scope  is  a  single  process  area).  The  upper  boundary  for  the  size  of  an 
organizational  unit  is  detennined  by  the  requirement  to  complete  Phase  II  of  the  appraisal  process 
within  90  days  (without  compromising  the  data  coverage  requirements  of  the  method). 

As  the  appraisal  sponsor  and  the  lead  appraiser  define  the  sampling  necessary  to  meet 
requirements  for  a  representative  sample  with  adequate  coverage  within  the  organizational  unit, 
the  definition  of  organizational  unit  may  need  to  be  adjusted  to  fit  the  cost,  resource  and  schedule 
constraints  for  the  appraisal. 

Basic  Units  and  Support  Functions 

The  sampling  process  used  in  SCAMPI  relies  on  a  key  distinction  between  groups  of  people  who 
perform  work  which  is  typically  visible  to  customers  (basic  units)  and  groups  of  people  who 
perform  work  that  enables  or  supports  the  work  which  is  ultimately  visible  to  the  customers  of  the 
organization  (support  functions).  The  Appraisal  Reference  Model  (e.g.,  CMMI-DEV,  CMMI- 
SVC,  CMMI-ACQ,  People  CMM)  used  will  determine  the  types  of  basic  units  or  support 
functions  which  are  appropriate  for  the  work  performed  in  the  organizational  unit.  When  multiple 
Appraisal  Reference  Models  are  used,  the  same  staff  may  perform  work  across  multiple 
representations.  Examples  of  basic  units  include  projects,  work  groups,  and  teams.  Examples  of 
support  functions  include  Quality  Assurance,  Configuration  Management,  Training  and 
Engineering  Process  Groups.  The  lead  appraiser  works  with  the  organization  to  understand  the 
basic  units  and  support  functions  into  which  people  are  organized  to  perform  work. 

Sampling  Factors 

Sampling  factors  serve  to  identify  meaningful  differences  in  the  conditions  under  which  work  is 
performed  in  the  organizational  unit.  Based  on  a  thorough  understanding  of  the  organization,  the 
lead  appraiser  determines  the  sampling  factors  that  define  different  clusters  of  process 
implementation  for  the  organization  unit.  Tiers  of  the  organization  chart  often  provide  an  initial 
view  of  these  potential  groupings.  The  Method  Definition  Document,  Section  1.1.4,  contains  a  list 
of  potential  sampling  factors  which  must  be  evaluated.  In  addition,  the  lead  appraiser  seeks 
information  about  other  potential  sampling  factors. 


Subgroups 

Sampling  factors  are  used  to  define  subgroups  in  the  organizational  unit.  Subgroups  consist  of  sets 
of  basic  units  which  share  the  attributes  identified  by  the  sampling  factors.  Subgroups  are  defined 
by  detennining  all  potential  combinations  for  each  value  of  the  sampling  factors.  In  the  example 
below  the  sampling  factors  ‘location’  and  ‘customer’  are  used  to  illustrate  the  formation  of 
subgroups  into  which  30  basic  units  are  classified: 
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Table  1 9:  Example  of  Sampling  Factors  -  Customer 


Commercial 

Government 

Customers 

Customers 

New  York 

7 

2 

Cincinnati 

5 

0 

Denver 

11 

5 

The  six  subgroups  defined  above  show  the  count  of  basic  units  in  each  possible  combination  of 
the  two  sampling  factors  used  in  the  illustration.  Notice  that  there  are  no  basic  units  in  Cincinnati 
that  have  govermnent  customers. 


Determine  Sample 

A  representative  sample  of  basic  units  is  determined  using  the  computation  defined  below. 
Sampling  formula: 


Minimum  number 
of  basic  units  to 
be  selected  from 
a  given  subgroup 


Number  of 
subgroups  X 


Number  oT 
basic  units  in  the 
given  subgroup 


Total  number 
of  basic  units 


The  result  of  this  computation  is  the  minimum  acceptable  sample  for  the  appraisal  with  at  least 
one  basic  unit  sampled  from  each  subgroup.  The  computed  value  of  the  formula  may  generate  a 
fractional  number.  If  the  computed  value  using  this  formula  is  less  than  1,  then  the  required 
number  of  basic  units  will  be  1.  Fractional  units  over  1  are  subject  to  normal  rounding  rules  (e.g. 
1.49  would  become  1;  1.5  would  become  2). 

Applying  the  formula  above  to  the  simple  example  introduced  earlier  yields  the  results  in  the  table 
below: 


Table  20:  Example  of  Sampling  Factors  -  Basic  Units 


Total  Number 

of  Basic  Units 

Basic  Units 
Sampled 

New  York,  Commercial 

7 

1 

New  York,  Government 

2 

1 

Cincinnati,  Commercial 

5 

1 

Denver,  Commercial 

11 

2 

Denver,  Government 

5 

1 
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The  lead  appraiser  and  sponsor  may  agree  on  the  involvement  of  additional  basic  units  as 
appropriate  (e.g.,  if  the  sponsor  chooses  to  include  specific  work  efforts  in  order  to  address 
expectations  of  customers  who  want  to  see  appraisal  results  for  work  delivered  under  a  given 
contract).  As  well,  the  discussion  between  the  sponsor  and  the  lead  appraiser  may  lead  to  reducing 
the  breadth  of  the  organizational  unit  involved  in  the  appraisal  —for  example,  omitting  the 
Cincinnati  location  altogether-  leading  to  a  smaller  scope  for  the  appraisal  activity.  This  fact 
would  be  documented  with  the  appraisal  results,  which  would  not  apply  to  the  status  of  the 
processes  used  by  basic  units  in  the  Cincinnati  location. 

Plan  Data  Collection 

Artifacts  and  affirmations  are  required  to  satisfy  the  data  coverage  rules  specified  in  the  MDD. 

The  initial  data  collection  plan  identifies  how  the  data  coverage  rules  will  be  satisfied.  This  initial 
plan  provides  an  added  level  of  detail  for  the  appraisal  sponsor  and  lead  appraiser  to  understand 
the  effort  required  to  perfonn  the  appraisal.  At  this  point  in  the  process,  before  data  collection  has 
begun,  the  organizational  unit  may  again  be  redefined  in  order  to  achieve  a  more  desirable  balance 
between  the  cost  of  the  appraisal  and  the  breadth  of  applicability  for  the  appraisal  results. 

Data  Coverage  Rules 

The  data  coverage  rules  defined  in  SCAMPI,  in  concert  with  the  sampling  process,  promote  a 
balance  between  breadth  and  depth  of  data  collection  during  an  appraisal.  These  rales  must  be 
interpreted  in  combination  to  define  the  data  collection  plan  for  an  appraisal,  rather  than  addressed 
one  rale  at  a  time.  In  most  applications  of  the  appraisal  method,  many  different  ways  to  meet  the 
set  of  data  coverage  rales  will  be  possible.  This  flexibility  allows  the  lead  appraiser  and  appraisal 
sponsor  to  collaborate  to  devise  an  efficient  sampling  and  data  collection  strategy,  where  cost- 
value  tradeoffs  can  be  made  within  an  objective  framework  that  promotes  credibility  of  the 
appraisal  results.  At  the  end  of  the  appraisal,  the  qualified  and  trained  appraisal  team  must  come 
to  consensus  on  the  adequacy  of  the  information  considered  to  render  decisions. 

The  sampling  process  sets  a  scope  for  this  decision  process,  and  the  data  coverage  rales  provide  a 
minimum  threshold  for  inputs  to  the  team  decisions.  Depending  on  what  is  found  during  the 
examination  of  the  required  data,  the  team  may  determine  that  additional  information  is  required 
to  make  a  fair  and  balanced  judgment.  For  example,  if  weaknesses  are  found  through  examination 
of  some  artifacts,  additional  affirmations  may  be  sought  to  ensure  the  team  is  interpreting  the 
artifacts  correctly.  Similarly,  if  affirmations  indicate  greater  strength  of  implementation  than  the 
initial  set  of  artifacts  support,  the  team  may  seek  other  artifacts  to  corroborate  the  information 
supplied. 

Introduction  to  Case  Studies 

The  three  case  studies  described  in  this  appendix  provide  illustrations  of  the  sampling  and  data 
coverage  concepts  in  order  to  foster  understanding.  These  case  studies  are  not  intended  to  provide 
specific  templates  to  be  emulated  in  practice;  rather  they  provide  examples  of  the  kinds  of 
analyses  that  are  to  be  employed  in  planning  a  SCAMPI  VI. 3.  As  summaries  of  real  situations, 
note  that  none  of  these  case  studies  reflect  the  entire  set  of  information  expected  to  be  considered 
by  appraisal  team  leaders  who  apply  the  process  described  in  the  MDD. 
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While  the  formats  of  the  three  case  studies  are  similar,  they  intentionally  present  various  levels  of 
detail  that  would  be  appropriate  given  the  focus  of  each  case  study.  Each  case  study  has  the 
following  major  section  headings: 

•  The  Organization  and  Initial  Identification  of  Sampling  Factors 

•  Identification  of  Subgroups 

•  Scoping  the  Organizational  Unit  and  Associated  Sample  Size 

•  Evaluating  the  Tradeoffs 

•  Apply  the  Data  Coverage  Requirements 

•  Case  Study  Summary 
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Case  1:  Large  Development  Organization 

The  Organization  and  Initial  Identification  of  Sampling  Factors 

The  appraisal  team  leader  has  started  to  work  with  a  large  aerospace  and  defense  corporation  to 
perform  a  Maturity  Level  2  appraisal.  The  organization  has  been  improving  process  for  over  1 0 
years.  Based  on  conversations  with  the  sponsor,  the  appraisal  team  leader  initially  determines  the 
organizational  unit  should  be  the  Blue  Business  Unit.  The  Blue  Business  Unit  is  comprised  of  30 
projects. 

The  documented  analysis  resulted  in  the  following  sampling  factors  with  their  associated  relevant 
values: 

•  Location  (Los  Angeles,  Dayton,  Other) 

•  Customer  (DOD,  Commercial) 

•  Size  (Large>15  people,  Small<15  people) 

•  Duration  (Long  >  lyr,  Short  <  lyr) 

The  potential  sampling  factors  “Organizational  Structure’  and  ‘Type  of  Work’  were  also 
evaluated,  and  found  not  to  play  an  important  role  in  the  organization.  Detailed  analysis  of  the 
organization  found  that  the  sampling  factors  ‘Location’  and  ‘Customer’  accounted  for  all 
differences  associated  with  these  other  potential  sampling  factors.  It  was  documented  that  the 
organizational  structure  was  partitioned  by  location,  and  the  type  of  work  was  completely 
accounted  for  by  the  customer.  That  is,  each  company  location  contained  only  one  part  of  the 
organizational  structure,  and  different  types  of  work  were  not  performed  for  a  given  customer. 

The  sampling  factor  ‘Duration’  was  added  to  the  sampling  process  because  the  threshold  of  1  year 
in  project  duration  was  found  to  have  a  notable  influence  on  the  process. 

Identification  of  Subgroups 

From  this  the  appraisal  team  leader  determines  the  number  of  subgroups  by  applying  the  sampling 
factors  to  the  projects  in  the  organizational  unit.  This  analysis  results  in  the  table  of  subgroups 
identified  below.  Note  that  there  are  24  possible  subgroups  given  the  sampling  factors  and 
associated  possible  values  shown  above. 
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Only  eight  subgroups  are  identified  below  because  all  of  the  organization’s  basic  units  exist  in 
only  these  eight  subgroups. 

Table  21:  Subgroups  Defined  by  Sampling  Factors 


Location 

Customer 

Size 

Duration 

#  Basic  Units 

LA 

Commercial 

Large 

Short 

2 

LA 

Commercial 

Small 

Short 

1 

LA 

DoD 

Large 

Long 

4 

LA 

DoD 

Small 

Long 

8 

Dayton 

DoD 

Large 

Long 

2 

Dayton 

DoD 

Small 

Long 

6 

Other 

DoD 

Large 

Long 

3 

Other 

DoD 

Small 

Long 

4 

30 

Scoping  the  Organizational  Unit  and  Associated  Sample  Size 

Several  different  scenarios  are  developed  to  show  the  impact  on  the  organizational  unit  and 
organizational  scope.  Refer  to  the  formula  in  MDD  Section  1.1.4  to  determine  the  “minimum 
number  of  basic  units  to  be  selected  from  a  given  subgroup. 
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OU:  Blue  Business  Unit 

(i.e.,  the  full  organizational  unit) 

Table  22:  Subgroups  and  Sample  Size  (Blue  Business  Unit) 


Subgroup  Attributes 
(Sampling  Factor  Values) 

Count  of 

Basic  Units 

Result  from 
Applying  the 
Formula 

Number  of  Basic 
Units  to  Sample 

LA,  Commercial,  Large,  Short 

2 

0.533 

1 

LA,  Commercial,  Small,  Short 

1 

0.267 

1 

LA,  DoD,  Large,  Long 

4 

1.067 

1 

LA,  DoD,  Small,  Long 

8 

2.133 

2 

Dayton,  DoD,  Large,  Long 

2 

0.533 

1 

Dayton,  DoD,  Small,  Long 

6 

1.600 

2 

Other,  DoD,  Large,  Long 

3 

0.800 

1 

Other,  DoD,  Small,  Long 

4 

1.067 

1 

10 
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OU:  Blue  Business  Unit  LA  and  Dayton  Locations 

(i.e.,  limiting  organizational  unit  to  two  locations  -  omitting  the  “other  location”) 
Table  23:  Subgroups  and  Sample  Size  (Blue  Business  Unit  LA  and  Dayton  Locations) 


Subgroup  Attributes 
(Sampling  Factor  Values) 

Count  of  Basic 
Units 

Result  from 
Applying  the 
Formula 

Number  of  Basic 
Units  to  Sample 

LA,  Commercial,  Large,  Short 

2 

0.522 

1 

LA,  Commercial,  Small,  Short 

1 

0.261 

1 

LA,  DoD,  Large,  Long 

4 

1.043 

1 

LA,  DoD,  Small,  Long 

8 

2.087 

2 

Dayton,  DoD,  Large,  Long 

2 

0.522 

1 

Dayton,  DoD,  Small,  Long 

6 

1.565 

2 

8 

OU:  Blue  Business  Unit  LA  and  Dayton  Locations/Large  Projects 

(i.e.,  limiting  organizational  unit  further  -  also  omitting  “small  projects”  from  the  organizational 
unit) 

Table  24:  Subgroups  and  Sample  Size  (Blue  Business  Unit  LA  and  Dayton  Locations/Large  Projects) 


Subgroup  Attributes 
(Sampling  Factor  Values) 

Count  of  Basic 
Units 

Result  from 
Applying  the 
Formula 

Number  of  Basic 
Units  to  Sample 

LA,  Commercial,  Large,  Short 

2 

0.750 

1 

LA,  DoD,  Large,  Long 

4 

1.500 

2 

Dayton,  DoD,  Large,  Long 

2 

0.750 

1 

4 
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Evaluating  the  Tradeoffs 

The  appraisal  sponsor  and  appraisal  team  leader  can  work  with  the  alternative  scenarios  above 
(and  others  as  they  wish)  to  evaluate  the  tradeoff  between  the  scope  and  the  magnitude  of  the 
appraisal  event.  Each  organizational  unit  definition  and  accompanying  organizational  scope 
alternative  would  present  a  different  scenario  in  planning  data  collection  for  the  appraisal  event.  A 
summary  the  alternatives,  associated  number  of  subgroups  and  number  of  samples  is  provided  in 
the  following  table. 


Table  25:  Summary  of  the  Blue  Business  Unit’s  Organizational  Unit  and  Scope  Alternatives 


Organizational 
Unit  Name  and 
Scenario 

Comments 

# 

Subgroups 

#  Sampled 
Basic 

Units 

The  Blue 

Business  Unit 

All  elements  of  the  Blue  Business  Unit  are  included, 
so  stakeholders  in  the  appraisal  outcome  will 
expect  the  results  to  apply  to  all  of  the  different 
types  of  work  done  within  the  Business  Unit. 

8 

10 

The  Blue 

Business  Unit/LA 
and  Dayton 
Locations  only 

The  scope  of  the  appraisal  would  be  limited  to  the 

LA  and  Dayton  locations.  Assertions  about  the 
results  of  this  appraisal  would  apply  only  to  the 
work  in  these  two  locations. 

6 

8 

The  Blue 

Business  Unit/LA 
and  Dayton 
Locations/Large 
Projects  only 

The  scope  of  the  appraisal  would  be  limited  to  the 
part  of  the  organization  that  conducts  large  projects 
in  the  LA  and  Dayton  locations.  Assertions  about 
the  results  of  this  appraisal  would  apply  only  to 
large  projects  in  these  two  locations. 

3 

4 

Applying  the  Data  Coverage  Requirements 

The  data  coverage  rules  are  applied  to  each  of  the  scenarios  developed  earlier.  Further 

assumptions  made  based  on  analyzing  the  organization  are  as  follows: 

•  A  centralized  QA  support  function  operates  at  each  location,  sharing  staff  and  using  location- 
specific  infrastructure  and  personnel.  The  function  is  applied  unifonnly,  with  shared  staff, 
common  templates,  and  reporting  expectations.  The  customer,  size  and  duration  of  the  project 
do  not  affect  the  behavior  of  the  staff  perfonning  the  work.  The  following  tables  demonstrate 
application  of  coverage  rule  3  for  support  functions.  At  least  one  sample  of  PPQA  is  required 
for  each  location. 

•  A  Project  Management  Office  was  established  to  manage  all  DoD  projects,  and  the 
management  structure  overseeing  those  projects  operates  as  a  cross-site  function.  Standard 
planning  and  monitoring  processes  are  implemented,  and  personnel  who  perform  that  work 
have  responsibility  for  multiple  DoD  projects.  The  location,  duration  and  size  of  the  project 
do  not  affect  the  behavior  of  the  staff  doing  planning  and  monitoring  on  DoD  projects.  In  this 
instance,  PMO  is  a  support  function  for  DoD  projects.  The  following  tables  demonstrate 
application  of  coverage  mle  3  for  support  functions.  At  least  one  sample  of  PP  and  PMC  is 
required  for  DoD  projects.  For  commercial  projects  however,  data  covering  PP  and  PMC  are 
required  from  each  subgroup  in  accordance  with  coverage  rule  1  for  basic  units. 
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•  Small  projects  share  a  common  infrastructure  for  configuration  management,  including  CM 
tools  and  persomiel  —  who  are  assigned  from  a  larger  pool.  For  large  projects,  project-specific 
infrastructure  and  staffing  are  established.  In  this  instance,  CM  is  a  support  function  for  small 
projects.  The  following  tables  demonstrate  application  of  coverage  rule  3  for  support 
functions.  At  least  one  sample  is  required  for  small  projects.  For  large  projects  however,  data 
covering  CM  is  required  from  each  subgroup  in  accordance  with  coverage  rule  1  for  basic 
units. 

•  DoD  projects  do  not  use  suppliers  —  as  a  matter  of  policy.  SAM  is  not  applicable  for  DoD 
projects. 

OU:  Blue  Business  Unit 

(i.e.,  the  full  business  unit) 

The  Data  Collection  Plan  depicted  below  conforms  to  the  rules,  with: 

•  One  unit  providing  PP  and  PMC  data  from  DoD  programs 

•  One  unit  each  providing  PPQA  data  from  the  LA  and  Dayton  locations 

Cells  containing  an  “x”  signify  a  requirement  to  collect  data  (affirmations  or  artifacts)  sufficient 

to  cover  the  process  area  represented  by  that  column,  for  the  basic  unit  represented  by  that  row. 

Cells  containing  an  “o”  signify  a  process  area  -  basic  unit  combination  for  which  data  are  not 

required,  according  to  the  minimum  coverage  requirements  of  the  method. 
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Table  26:  Subgroups  and  Sampled  Process  Areas  (Blue  Business  Unit) 


REQM 

PP 

PMC 

MA 

CM 

PPQA 

SAM 

Relevant  Sampling  Factors 

Sample 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Doc 

Aff 

LA,  Commercial,  Large,  Short 

1 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

LA,  Commercial,  Small,  Short 

1 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

LA,  DoD,  Large,  Long 

1 

X 

X 

X 

X 

X 

X 

X 

X 

LA,  DoD,  Small,  Long 

X 

X 

X 

X 

0 

0 

2 

0 

0 

0 

X 

0 

0 

Dayton,  DoD,  Large,  Long 

1 

X 

X 

X 

X 

X 

X 

Dayton,  DoD,  Small,  Long 

X 

X 

X 

X 

X 

X 

X 

X 

0 

0 

X 

X 

2 

0 

X 

0 

0 

0 

0 

Other,  DoD,  Large,  Long 

1 

X 

X 

X 

X 

X 

X 

X 

X 

Other,  DoD,  Small,  Long 

1 

X 

X 

X 

X 

0 

0 

X 

X 

10 

8 

9 

3 

3 

3 

3 

8 

9 

5 

5 

4 

4 

2 

2 

OU:  Blue  Business  Unit  LA  and  Dayton  Locations 

(i.e.,  limiting  organizational  unit  to  two  locations  -  omitting  the  “other  location”) 

Only  major  difference  is  the  omission  of  “other  location”  from  the  scope  of  the  organizational 
unit.  The  appraisal  results  would  apply  to  only  the  LA  and  Dayton  locations.  (Note  that  if  the 
process  did  not  vary  that  much  between  the  “other  locations”  and  LA/Dayton,  the  organizational 
unit  could  be  considered  to  be  the  full  organization  —  this  would  be  left  up  to  the  discretion  of  the 
appraisal  team  leader.) 

As  in  the  first  alternative,  the  Data  Collection  Plan  depicted  below  conforms  to  the  rules,  with: 

•  One  unit  providing  PP  and  PMC  data  from  DoD  programs 

•  One  unit  providing  PPQA  data  from  the  LA  location 

Cells  containing  an  “x”  signify  a  requirement  to  collect  data  (affirmations  or  artifacts)  sufficient 
to  cover  the  process  area  represented  by  that  column,  for  the  basic  unit  represented  by  that  row. 
Cells  containing  an  “o”  signify  a  process  area  —  basic  unit  combination  for  which  data  are  not 
required,  according  to  the  minimum  coverage  requirements  of  the  method. 


CMU/SEI-201 1-HB-001  |  220 


Table  21:  Subgroups  and  Sampled  Process  Areas  (LA  and  Dayton  Locations  Only) 


REQM 

PP 

PMC 

MA 

CM 

PPQA 

SAM 

Relevant  Sampling  Factors 

Sample 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

LA,  Commercial,  Large, 

Short 

1 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

LA,  Commercial,  Small, 

Short 

1 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

LA,  DoD,  Large,  Long 

1 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

LA,  DoD,  Small,  Long 

2 

X 

X 

X 

X 

0 

0 

X 

0 

0 

0 

0 

0 

Dayton,  DoD,  Large,  Long 

1 

X 

X 

X 

X 

X 

X 

X 

X 

Dayton,  DoD,  Small,  Long 

2 

X 

X 

X 

X 

0 

0 

0 

0 

X 

0 

0 

0 

8 

7 

6 

3 

3 

3 

3 

7 

6 

4 

4 

2 

2 

2 

2 

OU:  Blue  Business  Unit  LA  and  Dayton  Locations/Large  Projects 

(i.e.,  limiting  organizational  unit  further  -  also  omitting  “small  projects”  from  organizational  unit) 

This  alternative  further  limits  the  organizational  unit  to  large  projects.  Consistent  with  the 
alternative  presented  above,  the  Data  Collection  Plan  depicted  below  conforms  to  the  rules,  with: 

•  One  unit  providing  PP  and  PMC  data  from  DoD  programs 

•  One  unit  providing  PPQA  data  from  the  LA  location 

Cells  containing  an  “x”  signify  a  requirement  to  collect  data  (affirmations  or  artifacts)  sufficient 
to  cover  the  process  area  represented  by  that  column,  for  the  basic  unit  represented  by  that  row. 
Cells  containing  an  “o”  signify  a  process  area  —  basic  unit  combination  for  which  data  are  not 
required,  according  to  the  minimum  coverage  requirements  of  the  method. 
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Table  28:  Subgroups  and  Sampled  process  areas  (LA  and  Dayton  Locations/Large  Projects  Only) 


REQM 

pp 

PMC 

MA 

CM 

PPQA 

SAM 

Relevant  Sampling  Factors 

Sample 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

LA,  Commercial,  Large, 

Short 

1 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

LA,  DoD,  Large,  Long 

X 

X 

X 

X 

X 

X 

X 

X 

2 

0 

X 

X 

X 

X 

X 

0 

0 

0 

0 

Dayton,  DoD,  Large,  Long 

1 

X 

X 

X 

X 

X 

X 

X 

X 

4 

3 

4 

2 

2 

2 

2 

3 

3 

3 

3 

2 

2 

1 

1 

Case  1  (Large  Development  Organization)  Summary 

Under  the  rules  of  SCAMPI  VI. 3,  the  Blue  Business  Unit  will  be  able  to  explore  many 
alternatives  pertaining  to  the  definition  of  the  organizational  unit  and  associated  organizational 
scope,  as  appropriate.  This  will  allow  the  Blue  Business  Unit  to  conduct  an  appraisal  that  satisfies 
the  needs  of  the  appraisal  sponsor  while  assuring  the  quality  of  the  appraisal  result. 


CMU/SEI-201 1-HB-001 


222 


Case  2:  Services  Organization 

The  Organization  and  Initial  Identification  of  Sampling  Factors 

The  appraisal  team  leader  is  working  with  a  well-established  firm  providing  staff-augmentation 
services  for  a  diverse  set  of  software  and  systems  engineering  programs  in  the  defense  and 
intelligence  industries.  The  organization  has  been  engaged  in  process  improvement  for  a  number 
of  years,  and  is  now  interested  in  establishing  a  Maturity  Level  2  benchmark  using  CMMI-SVC  to 
enhance  their  reputation  among  their  customer  base.  Following  conversations  with  the  sponsor, 
the  appraisal  team  leader  initially  determines  the  appropriate  organizational  unit  should  include 
the  entire  company  —  because  of  the  integrated  nature  of  the  processes  and  the  company- wide 
focus  of  the  improvement  program.  The  basic  unit  for  organizing  the  work  in  this  organization  is 
‘the  contract’  —  because  each  contract  has  the  potential  to  present  some  unique  needs  and 
constraints  on  various  processes  used  in  allocating  staff  and  managing  their  ongoing  service. 
Analysis  of  the  organization  results  in  the  following  sampling  factors  (with  relevant  value): 

•  Assignment  Duration  (Short-Term  =  less  than  6  months;  Medium-Term  =  6  months  to  1  year; 
Long-Term  =  more  than  1  year;  Semi-Permanent  =  no  end-date  specified) 

•  Technical  Competency  Required  (Requirements  Specification  and  Analysis;  Systems 
Feasibility  Study  and  Proof  Of  Concept/Prototype  Development;  Software 
Engineering/Development;  Systems  Integration/Testing;  Documentation/Training 
Development) 

•  Program  Domain  (Communications,  Command,  Control  and  Intelligence;  Advance  Weapons 
Systems;  Aeronautical  Engineering;  Logistics  and  Supply  Chain  Management)  (Note  this  is 
considered  an  example  of  the  “customer”  sampling  factor) 

•  Security/Classification  Requirements  (Non-Classified  Environment;  Classification-Level  1; 
Classification-Level  2;  Unique  Classification  Requirement) 

The  potential  sampling  factors  which  the  SCAMPI  method  requires  to  be  analyzed  (refer  to 
Section  1.1.4)  were  ruled  out  from  consideration  based  on  the  analysis  summarized  below: 

•  Location:  the  service  being  delivered  by  the  company  is  executed  from  a  single  centralized 
location,  though  the  assignments  of  the  technical  staff  take  them  all  around  the  world. 

•  Customer:  the  combination  of  technical  competency  required  and  program  domain  define  the 
range  of  customers  served  by  the  company.  Defining  subgroups  using  ‘customer’  would  be 
redundant  with  these  two  sampling  factors. 

•  Size:  the  size  of  the  programs  being  staffed  overlap  (in  terms  of  categories)  with  the 
assignment  duration. 

•  Organizational  Structure:  there  is  no  effect  on  the  behavior  or  practices  used  by  the  assigned 
staff  that  can  be  traced  back  to  an  aspect  of  the  organizational  structure  of  the  company 
providing  staff  augmentation. 

•  Type  of  Work:  the  sampling  factor  ‘program  domain’  essentially  represents  this  sampling 
factor  as  described  in  the  MDD,  but  the  label  ‘program  domain’  is  more  readily  understood  in 
the  organization. 
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Identification  of  Subgroups 

During  the  process  of  determining  the  implementation  of  support  functions,  a  greater  degree  of 
understanding  for  sampling  factors  is  established.  The  lead  appraiser  learns  that  differences  in  the 
performance  of  technical  work  may  have  been  over-emphasized  in  determining  sampling  factors 
that  relate  to  the  performance  of  the  CMMI-SVC-related  work  within  the  organization. 
Consequently,  the  following  two  sampling  factors  have  been  modified: 

•  Assigmnent  Duration:  Two  levels  have  been  combined  resulting  in  three  values  (Short-Term 
=  less  than  6  months;  Medium-Term  =  6  months  to  1  year;  Long-Term  =  more  than  1  year). 
The  processes  that  govern  staff  assignments  on  a  semi-permanent  basis  do  not  differ 
substantially  from  those  used  to  assign  staff  on  a  long-term  basis.  All  semi-permanent 
assignments  have  last  more  than  1  year.  Differences  in  the  content  of  the  service  level 
agreement  for  these  contracts  do  not  impact  the  practices  used  to  identify,  assign,  and  manage 
the  service  level  agreement. 

•  Security/Classification  Requirements:  Two  levels  have  been  combined  resulting  in  three 
values  (Non-Classified  Enviromnent;  Classification-Level  1  or  2;  Unique  Classification 
Requirement).  The  qualifications  of  each  staff  member  are  identified  with  respect  to  4  levels 
of  security  clearance  (with  the  first  level  signifying  that  the  staff  holds  no  clearance  of  any 
type).  Each  assignment  filled  by  a  staff  member  is  categorized  in  the  same  manner.  The 
processes  associated  with  the  ‘middle  two’  classification  levels  have  no  meaningful 
difference.  In  addition,  most  staff  assigned  to  positions  requiring  “Classification-Level  1” 
have  a  higher  level  of  clearance. 

The  appraisal  team  leader  determines  the  relevant  subgroups  by  applying  the  sampling  factors  to 
the  staff  augmentation  contracts  in  the  organizational  unit.  This  analysis  is  documented  in  the 
table  below.  The  table  captures  the  diversity  of  the  work  in  the  organizational  unit,  in  terms  of  the 
differences  among  contracts  that  are  thought  to  lead  to  potential  differences  in  process 
implementation.  The  subgroups  below  represent  different  potential  challenging  conditions 
(process  contexts)  that  may  at  times  test  the  robustness  of  the  policies  and  practices  of  the 
organization.  Note  that  there  are  180  possible  subgroups  given  the  sampling  factors  and 
associated  possible  values  shown  above. 
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Only  1 1  subgroups  are  identified  below  because  all  of  the  organization’s  basic  units  exist  in  only 
these  1 1  subgroups. 

Table  29:  Subgroups  Defined  by  Sampling  Factors 


Duration 

Technical 

Competency 

Domain 

Classification 

#  Basic  Units 

Short 

Requirements 

C3I 

None 

1 

Short 

Requirements 

Weapons 

None 

1 

Short 

Feasibility 

Log 

None 

1 

Short 

Document 

Log 

None 

2 

Long 

Integration 

Aero 

None 

3 

Long 

Software 

C3I 

1  or  2 

8 

Medium 

Requirements 

C3I 

Unique 

3 

Medium 

Feasibility 

Weapons 

1  or  2 

2 

Medium 

Feasibility 

Aero 

1  or  2 

2 

Long 

Feasibility 

Weapons 

Unique 

3 

Long 

Software 

C3I 

Unique 

4 

30 

Scoping  the  Organizational  Unit  and  Associated  Sample  Size 

Working  with  the  sponsor,  the  lead  appraiser  develops  different  scenarios  to  show  how  alternative 
definitions  of  the  organizational  unit  lead  to  various  samples  that  make  up  the  organizational 
scope.  Refer  to  the  formula  in  MDD  Section  1.1.4  to  determine  the  minimum  number  of  basic 
units  to  be  selected  from  a  given  subgroup. 
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OU:  The  Whole  Company 

(i.e.,  all  contracts) 

Table  30:  Subgroups  and  Sample  Size  (The  Whole  Company) 


Subgroup  Attributes 
(Sampling  Factor  Values) 

Count  of 
Basic 

Units 

Result  from 
Applying  the 
Formula 

Number  of 
Basic  Units  to 
Sample 

Short,  Requirements,  C3I,  No  Classification 

1 

0.333 

1 

Short,  Requirements,  Weapons,  No  Classification 

1 

0.333 

1 

Short,  Feasibility,  Logistics,  No  Classification 

1 

0.333 

1 

Short,  Documentation,  Logistics,  No  Classification 

2 

0.667 

1 

Long,  Integration,  Aerospace,  No  Classification 

3 

1.000 

1 

Long,  Software,  C3I,  Classification  1  or  2 

8 

2.667 

3 

Medium,  Requirements,  C3I,  Unique  Classification 

3 

1.000 

1 

Medium,  Feasibility,  Weapons,  Classification  1  or  2 

2 

0.667 

1 

Medium,  Feasibility,  Aerospace,  Classification  1  or  2 

2 

0.667 

1 

Long,  Feasibility,  Weapons,  Unique  Classification 

3 

1.000 

1 

Long,  Software,  C3I,  Unique  Classification 

4 

1.333 

1 

13 
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OU:  C3  I  Contracts  Only 

(i.e.,  limiting  organizational  unit  to  only  C3I  contracts) 

Table  31:  Subgroups  and  Sample  Size  (The  Red  Company/C3l  Contracts) 


Subgroup  Attributes 
(Sampling  Factor  Values) 

Count  of 
Basic 

Units 

Result  from 
Applying  the 
Formula 

Number  of 
Basic  Units 
to  Sample 

Short,  Requirements,  C3I,  No  Classification 

1 

0.250 

1 

Long,  Software,  C3I,  Classification  1  or  2 

8 

2.000 

2 

Medium,  Requirements,  C3I,  Unique  Classification 

3 

0.750 

1 

Long,  Software,  C3I,  Unique  Classification 

4 

1.000 

1 

5 

OU:  Non-Classified  Contracts  Only 

(i.e.,  limiting  organizational  unit  to  exclude  classified  contracts) 

Table  32:  Subgroups  and  Sample  Size  (The  Red  Company/Non-Classified  Contracts) 


Subgroup  Attributes 
(Sampling  Factor  Values) 

Count  of 
Basic 

Units 

Result  from 
Applying  the 
Formula 

Number  of 
Basic  Units 
to  Sample 

Short,  Requirements,  C3I,  No  Classification 

1 

0.625 

1 

Short,  Requirements,  Weapons,  No  Classification 

1 

0.625 

1 

Short,  Feasibility,  Logistics,  No  Classification 

1 

0.625 

1 

Short,  Documentation,  Logistics,  No  Classification 

2 

1.250 

1 

Long,  Integration,  Aerospace,  No  Classification 

3 

1.875 

2 

6 
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Evaluating  the  Tradeoffs 

The  appraisal  sponsor  and  appraisal  team  leader  work  with  the  alternative  scenarios  above  (and 
others  as  deemed  relevant)  to  evaluate  the  tradeoff  between  the  scope  and  the  magnitude  of  the 
appraisal  event. 

Each  organizational  unit  definition  and  accompanying  organizational  scope  presents  a  different 
scenario  in  planning  data  collection  for  the  appraisal  event.  As  well,  each  candidate  definition  of 
organizational  unit  leads  to  a  different  scope  of  generalization  which  can  be  made  with  appraisal 
results.  The  table  below  summarizes  these  differences  at  a  high  level: 


Table  33:  Summary  of  Organizational  Unit  and  Scope  Alternatives 


Organizational 
Unit  Name  and 
Scenario 

Comments 

# 

Subgroups 

# 

Samples 

The  Whole 
Company 

All  elements  of  the  organizational  unit  are  included, 
so  stakeholders  in  the  appraisal  outcome  will  expect 
the  results  to  apply  to  all  of  the  different  types  of  work 
done  within  the  organization. 

11 

13 

C3I  Contracts 

Only 

If  the  organization  is  competing  for  staff 
augmentation  contracts  in  this  domain,  a  benchmark 
that  applies  only  to  their  C3l-related  portfolio  of  work 
might  be  an  ideal  scoping  for  the  appraisal. 

4 

5 

Un-Classified 
Contracts  Only 

If  the  classification  restrictions  associated  with  the 
staffing  assignments  limits  the  accessibility  of  data, 
then  an  appraisal  focused  on  only  the  un-classified 
contracts  may  present  a  simpler  event  from  a 
logistics  perspective. 

5 

6 

Applying  the  Data  Coverage  Requirements 

The  data  coverage  rules  are  applied  to  each  scenario,  to  illustrate  the  ramifications  of  the  scoping 
and  sampling  decisions  for  the  magnitude  of  the  appraisal  event.  The  numbers  presented  represent 
minimum  acceptable  sampling  values  for  consideration.  In  order  to  assert  that  the  data  collected  is 
sufficient  to  represent  the  organization,  the  lead  appraiser  must  use  professional  judgment  to 
ensure  no  sampling  factors  -or  other  unforeseen  influences  on  the  process  context—  have  been 
overlooked. 
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In  addition  to  the  sampling  factors  identified  above,  the  following  information  relating  to  the 

allocation  of  support  functions  within  the  organization  is  documented  during  planning. 

•  A  highly  secure  Configuration  Management  support  function  has  been  established  for 
managing  assets  related  to  on-going  contracts  as  well  as  the  staff  and  their  capabilities.  Assets 
maintained  include  Proposals  and  Bids  for  classified  work,  as  well  as  resumes  and  personnel 
files  for  those  who  perform  this  work.  The  following  tables  demonstrate  application  of 
coverage  rule  1  for  support  functions.  At  least  one  sample  of  CM  is  required  across  the 
organizational  unit. 

•  The  organization’s  measurement  program  has  been  defined  ‘across  the  organization.’  It  is 
staffed  by  a  group  of  professionals  that  (collectively)  have  experience  with  all  types  of  work 
done  in  the  organization.  The  following  tables  demonstrate  application  of  coverage  rule  1  for 
support  functions.  At  least  one  sample  of  MA  is  required  across  the  organizational  unit. 

•  A  Project  Management  Office  (PMO)  structure  is  defined  to  separate  management  of 
classified  and  unclassified  work.  All  activities  relating  to  Work  Planning  and  Work 
Monitoring  and  Control  are  handled  by  a  centralized  function  —  one  for  unclassified  work, 
and  a  separate  one  for  classified  work.  The  following  tables  demonstrate  application  of 
coverage  rule  3  for  support  functions.  A  sample  of  WP  and  WMC  are  required  from  each 
instance  of  PMO,  one  for  classified  and  one  for  unclassified  work. 

•  All  other  ML2  process  areas  (PPQA,  REQM,  SD,  SAM)  have  potentially  ‘contract-unique 
implementations.’  The  tables  below  demonstrate  coverage  rule  1  for  basic  units.  For  the 
subgroup  with  three  sampled  basic  units,  coverage  rules  2  and  3  apply  as  well. 
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OU:  The  Whole  Company 

(i.e.,  all  contracts) 

Including  all  basic  units  and  support  functions  in  the  organization,  the  data  collection  planning 
begins  with  the  matrix  depicted  below.  The  coverage  rules  guide  more  detailed  data  collection 
planning.  Note  that  this  will  affect  the  number  of  pieces  of  data  required  for  the  sixth  subgroup  in 
the  table  —  for  the  columns  labeled  PPQA,  REQM,  SD,  and  SAM  -  because  3  units  are  sampled 
for  this  subgroup. 

Table  34:  Subgroups  and  Sampled  Process  Areas  (The  Red  Company) 


CM 

MA 

WMC 

WP 

PPQA 

REQM 

SD 

SAM 

Relevant  Sampling  Factors 

Sample 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Doc 

Aff 

Short,  REQ,  C3I,  Non-Class 

1 

X 

X 

X 

X 

X 

X 

X 

X 

Short,  REQ,  Weap,  Non-Class 

1 

X 

X 

X 

X 

X 

X 

X 

X 

Short,  Feas,  LOG,  Non-Class 

1 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Short,  DOC,  LOC,  Non-Class 

1 

X 

X 

X 

X 

X 

X 

X 

X 

Long,  INT,  Aero,  Non-Class 

1 

X 

X 

X 

X 

X 

X 

X 

X 

0 

0 

0 

0 

X 

0 

0 

0 

Long,  SW,  C3I,  Class  1  or  2 

3 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

0 

0 

X 

X 

0 

0 

0 

0 

Med,  REQ,  C3I,  Unique-Class 

1 

X 

X 

X 

X 

X 

X 

X 

X 

Med,  Feas,  Weap,  Cass  1  or  2 

1 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Med,  Feas,  Aero,  Class  1  or  2 

1 

X 

X 

X 

X 

X 

X 

X 

X 

Lng,  Feas,  Weap,  Unique-Class 

1 

X 

X 

X 

X 

X 

X 

X 

X 

Lng,  SW,  C3I,  Unique-Class 

1 

X 

X 

X 

X 

X 

X 

X 

X 

13 

1 

1 

1 

1 

2 

2 

2 

2 

11 

11 

12 

12 

12 

11 

11 

11 

The  columns  associated  with  CM  and  MA  process  areas  are  depicted  as  single  cells  because  these 
practices  are  performed  by  support  functions,  and  not  by  basic  units.  This  implies  that  an  appraisal 
team  examining  the  practices  associated  with  these  process  areas  would  examine  data  that  applies 
equivalently  across  all  basic  units  in  the  organizational  unit.  This  is  not  to  say  that  data  tracing  to 
individual  basic  units  would  not  be  examined. 
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In  contrast,  the  columns  for  WMC  and  WP  process  areas  are  depicted  with  cells  for  two  different 
implementations  —  one  for  non-classified  contracts,  and  the  other  for  classified  contracts.  This 
reflects  the  organization’s  choice  to  define  two  different  PMO  functions  for  the  two  types  of 
contracts  (non-classified  vs.  classified). 

OU:  C3  I  Contracts  Only 

(i.e.,  limiting  organizational  unit  to  only  C3I  contracts) 

Including  only  the  basic  units  tied  to  C3I  contracts  and  associated  support  functions  in  the 
organization,  the  data  collection  planning  begins  with  the  matrix  depicted  below.  Again,  the 
coverage  and  corroboration  rules  affect  the  number  of  pieces  of  data  required  for  the  row  labeled 
G6  -  for  the  columns  labeled  PPQA,  REQM,  SD,  and  SAM  -  because  two  units  are  sampled  for 
this  subgroup.  All  other  cells  in  the  matrix  would  have  at  least  one  item  of  data  (Art  =  Artifacts, 
Aff  =  Affirmations). 


Table  35:  Subgroups  and  Sampled  Process  Areas  (The  Red  Company/C3l  Contracts) 


CM 

MA 

WMC 

WP 

PPQA 

REQM 

SD 

SAM 

Relevant  Sampling  Factors 

Sample 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Doc 

Aff 

Short,  REQ,  C3I,  Non-Class 

1 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Long,  SW,  C3I,  Class  1  or  2 

0 

0 

0 

0 

X 

0 

0 

0 

2 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Med,  REQ,  C3I,  Unique-Class 

1 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Lng,  SW,  C3I,  Unique-Class 

1 

X 

X 

X 

X 

X 

X 

X 

X 

5 

1 

1 

1 

1 

2 

2 

2 

2 

4 

4 

4 

4 

5 

4 

4 

4 
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OU:  Non-Classified  Contracts  Only 

(i.e.,  limiting  organizational  unit  to  exclude  classified  contracts) 

Including  only  the  basic  units  tied  to  non-classified  contracts  and  associated  support  functions  in 
the  organization,  the  data  collection  planning  begins  with  the  matrix  depicted  below.  In  this  case, 
the  coverage  and  corroboration  rules  affect  the  number  of  pieces  of  data  required  for  the  row 
labeled  “Long,  INT,  Aero,  Non-Class”  -  for  the  columns  labeled  PPQA,  REQM,  SD,  and  SAM  - 
because  two  units  are  sampled  for  this  subgroup. 

In  the  context  of  non-classified  contracts,  this  subgroup  has  a  larger  number  of  basic  units  than  the 
other  subgroups,  so  the  minimum  sample  size  is  larger.  All  other  cells  in  the  matrix  would  have  at 
least  one  item  of  data  (Art  =  Artifacts,  Aff  =  Affirmations). 

Table  36:  Subgroups  and  Sampled  Process  Areas  (The  Red  Company/Non-Classified  Contracts) 


CM 

MA 

WMC 

WP 

PPQA 

REQM 

SD 

SAM 

Relevant  Sampling  Factors 

Sample 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Doc 

Aff 

Short,  REQ,  C3I,  Non-Class 

1 

X 

X 

X 

X 

X 

X 

X 

X 

Short,  REQ,  Weap,  Non-Class 

1 

X 

X 

X 

X 

X 

X 

X 

X 

Short,  Feas,  LOG,  Non-Class 

1 

X 

X 

X 

X 

X 

X 

X 

X 

Short,  DOC,  LOC,  Non-Class 

1 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Long,  INT,  Aero,  Non-Class 

0 

0 

X 

0 

0 

0 

0 

0 

2 

X 

X 

X 

X 

X 

X 

X 

X 

6 

1 

1 

1 

1 

1 

1 

1 

1 

5 

5 

6 

5 

5 

5 

5 

5 

Case  2  (Services  Organization)  Summary 

Under  the  mles  of  SCAMPI  VI  .3,  the  sponsor  for  this  company  will  be  able  to  explore  many 
different  alternatives  pertaining  to  the  definition  of  the  organizational  unit  and  associated 
organizational  scope,  as  appropriate.  This  will  allow  the  company  to  conduct  an  appraisal  that 
satisfies  the  needs  of  the  appraisal  sponsor  while  assuring  the  quality  of  the  appraisal  result. 
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Case  3:  Small  Development  Organization 

The  Organization  and  Initial  Identification  of  Sampling  Factors 

The  appraisal  team  leader  has  started  to  work  with  a  small  organization  that  is  working  to  achieve 
CMMI-DEV  Maturity  Level  3.  The  company  is  comprised  of  approximately  125  staff,  with  an 
engineering  workforce  of  80  maintaining  three  very  large  information  systems.  They  were  a 
fledgling  start  up  20  years  ago  when  five  hard-working  engineers  developed  a  niche  product  to 
service  a  government  program  dispensing  funds  through  the  Small  Business  Administration  to 
other  small  companies.  As  the  system  grew  to  serve  local  offices  in  municipalities  throughout  the 
US  and  a  wider  range  of  recipients,  the  company  expanded  the  sphere  of  their  work.  The  system 
that  launched  the  company  is  now  an  established  product  in  the  industry.  The  company  has 
ventured  into  developing  new  systems  based  on  the  architecture  of  their  flagship  product.  This 
success  has  brought  the  attention  of  the  major  corporate  player  in  the  region,  who  is  working  on  a 
buy-out  of  the  company.  Management  believes  a  successful  CMMI-based  process  improvement 
program  will  be  appealing  to  the  potential  buyer,  as  they  compete  for  contracts  in  a  market  where 
these  credentials  are  highly  valued. 

The  engineering  practices  used  on  projects  —  two  to  four  annual  releases  for  each  system  and 
“special  projects”  as  needed  -  tend  to  fall  into  two  different  categories.  The  flagship  product  is 
primarily  a  mainframe,  back-office,  batch-oriented  system  written  in  Cobol73.  While  training  and 
tools  have  greatly  influenced  the  support  environment  used  to  maintain  this  system,  the  testing 
and  operating  platforms  tend  to  differ  substantially  from  the  other  two  (more  modem)  systems. 
This  leads  to  notable  differences  in  processes  relating  to  Technical  Solution,  Verification, 
Validation,  Product  Integration  as  well  as  Integrated  Project  Management  process  areas.  The  two 
newer  systems  have  adopted  modern  design  methodologies,  higher  level  programming  languages 
and  more  modem  tools  that  integrate  easily  with  the  development  and  testing  environments  they 
use. 

Through  the  hard  work  of  the  EPG,  as  well  as  many  long  working  sessions  with  first  line 
managers  and  senior  engineers,  a  common  approach  to  managing  releases/projects  and  providing 
support  functions  has  been  established.  A  well-established  lifecycle  model  had  existed  for  over  a 
decade,  and  common  approaches  to  planning  and  monitoring  releases  is  in  place.  Disciplined 
approaches  to  configuration  and  release  management  are  supported  by  a  company-wide  quality 
assurance  and  measurement  system.  While  the  expertise  required  to  develop  and  maintain  good 
requirements  differs  according  to  the  history  of  the  customer-base  (and  the  experience  of  the  staff) 
the  fundamental  approach  is  the  same  across  the  company. 

The  documented  analysis  resulted  in  the  following  sampling  factors  with  their  associated  relevant 
values: 

•  Type  of  Project/Work  (Release,  Special  Project):  There  is  a  distinction  made  between  regular 
releases  of  systems,  and  special  projects.  Special  projects  are  carried  out  to  handle 
enhancements,  fixes,  or  special  features  out  of  cycle.  Changes  in  regulatory  constraints,  major 
security  threats,  or  significant  defect  reports  may  initiate  a  special  project. 

These  tend  to  be  smaller  in  scope  and  shorter  in  duration.  While  the  fundamental  elements  of 
the  development  and  management  processes  are  still  used  on  the  special  projects,  there  is  a 
distinct  “tailoring-down”  of  the  activities  to  accommodate  a  shorter  time  window.  Everything 
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from  requirements  development  to  release  management  -  and  the  project  planning  and 
oversight  that  occurs  in  between  —  is  done  with  abbreviated  timetables  and  a  different  level  of 
formality. 

•  Management  Structure  (Flagship  Division,  Other  Divisions). The  work  done  on  the  flagship 
product  has  provided  a  great  deal  of  opportunity  for  advancement  of  the  staff  in  that  division. 
The  new  divisions  were  seeded  with  the  innovative  members  of  that  division.  Consequently, 
there  is  a  larger  management  structure  —  accompanied  at  times  by  an  unreasonable  allegiance 
to  the  status  quo  —  in  that  division.  The  other  two  divisions  are  more  amenable  to  the 
introduction  of  new  practices,  and  therefore  able  to  institutionalize  improvements  in  a  shorter 
timeframe.  In  particular,  the  acceptance  of  new  ways  of  performing  the  practices  in  Decision 
Analysis  and  Resolution,  as  well  as  Risk  Management  process  area  may  be  an  issue. 

•  Technology  Base  (Legacy  Technology,  Modern  Technology):  The  legacy  system  identified  as 
the  flagship  product  leads  to  notable  differences  in  the  implementation  of  some  process  areas 
-  including  Technical  Solution,  Verification,  Validation,  Product  Integration  as  well  as 
Integrated  Project  Management. 

•  Depth  of  Product  History  (Flagship  Division,  Other  Divisions):  Work  on  the  flagship  system 
over  the  years  has  resulted  in  a  wealth  of  historical  information  which  is  not  yet  available  for 
the  other  products.  While  the  divisions  make  an  effort  to  share  historical  experience  and  data, 
their  usefulness  is  viewed  to  be  limited.  The  depth  of  experience  also  leads  to  a  greater 
preference  to  rely  on  intuition  and  engineering  judgment  at  times.  This  has  affected  the 
acceptance  of  Risk  Management  and  Decision  Analysis  and  Resolution. 

•  Though  not  described  in  detail  here,  other  potential  sampling  factors  (including  the  required 
list  in  Section  1.1.4)  were  analyzed  in  detail,  and  the  results  of  that  analysis  were  documented 
by  the  lead  appraiser  in  the  appraisal  plan. 

Identification  of  Subgroups 

Subgroups  within  the  organizational  unit:  The  table  below  summarizes  the  current  inventory  of 

releases  and  projects  in  the  organization,  according  to  the  organizational  structure.  The  sampling 

factors  identified  and  the  subsequent  analysis  performed  on  them  will  lead  to  the  identification  of 

subgroups  and  the  basic  units  they  contain. 

Table  37:  Number  of  Projects  in  the  Green  Company  Divisions 


Division 

Work  Type 

Release  or  Project  Name 

Count 

Flagship  Division 

Release 

Rel2010.3,  Rel2011.1 

2* 

Special  Project 

- 

0 

New  Division  A 

Release 

Rel2010.3,  Rel2010.4,  Rel2011.1 

3* 

Special  Project 

DataWarehouse201 1 

1 

New  Division  B 

Release 

Rel2010.2,  Rel2011.1 

2* 

Special  Project 

SecurityReportingA  B  C, 

FirmwareTester20 1 1 

2 
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*  At  any  given  time,  there  will  be  more  than  one  release  in  progress,  at  different  points  in  the 
lifecycle.  These  are  each  counted  as  one  basic  unit. 

There  may  be  times  when  no  special  projects  are  underway  in  a  given  division.  The  definition  of 
the  subgroups  is  provided  below. 

Table  38:  Definition  of  Subgroups 


Type  of  Work 

Management  Structure 

Technology 

Base 

Depth  of 
History 

#  Basic 
Units 

Release 

Flagship  Division 

Legacy 

Deep 

1 

Release 

New  Divisions 

Next  Gen 

New 

2 

Special  Projects 

New  Divisions 

Next  Gen 

New 

3 

6 

Scoping  the  Organizational  Unit  and  Associated  Sample  Size 

Follow-up  discussions  with  the  Engineering  Process  Group  leader,  interviews  of  first  line 
managers,  and  examination  of  past  appraisal  records  reveals  some  additional  information: 

•  Overlapping  Releases 

Given  the  strategy  for  rolling  releases,  at  any  given  point  in  time  recent  examples  of 
artifacts  and  activities  for  each  lifecycle  phase  are  available.  Flowever,  only  during 
certain  times  (just  after  a  release)  will  those  current  examples  all  relate  to  the  same 
release. 

All  releases  underway  at  the  current  time  are  based  on  the  same  version  of  the  company 
process.  No  major  process  changes  —  other  than  minor  revisions  to  templates  or  work 
instructions  -  have  occurred  since  the  completion  of  the  last  release  in  each  division. 

The  staff  working  on  releases  in  each  division  is  drawn  from  the  same  pool  -  the 
assignments  to  different  releases  can  potentially  change  from  day  to  day. 

The  EPG  suggests  that  it  is  reasonable  to  pool  all  the  releases  within  each  division  — 
rather  than  considering  them  separately  for  the  purpose  of  the  appraisal.  This  was  the 
approach  taken  in  the  last  class  B  appraisal  perfonned  in  the  company. 

•  Special  Projects 

At  any  given  time,  there  are  typically  two  to  six  special  projects  underway  in  the 
organization.  At  present  there  are  three  going  on. 

Refer  to  the  formula  in  MDD  Section  1.1.4  to  determine  the  minimum  number  of  basic  units  to  be 
selected  from  a  given  subgroup.  The  result  of  applying  this  computation  is  shown  in  the  table 
below. 
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Table  39:  Subgroups  and  Number  of  Sampled  Basic  Units 


Subgroup  Attributes 
(Sampling  Factor  Values) 

#  Basic 
Units 

Result  from 
|Applying  the 
Formula 

Number  of 
Basic  Units  to 
Sample 

Releases  in  Flagship  Division 

1 

0.500 

1 

Releases  in  New  Divisions 

2 

1.000 

1 

Special  Projects  in  New  Divisions 

3 

1.500 

2 

4 

*  There  is  no  sampling  factor  for  “release  id”  and  rather  than  treating  each  release  as  a  basic  unit, 
the  set  of  releases  within  a  division,  supported  by  a  common  pool  of  staff  in  the  division,  is  treated 
as  a  single  basic  unit.  Therefore,  each  division  is  a  basic  unit  in  this  case. 


Evaluating  the  Tradeoffs 

Although  less  likely  than  in  the  first  two  case  studies,  Green  Company  can  also  perform  an 
analysis  of  alternative  scenarios  relative  to  the  organization  unit  and  scope.  Performing  this 
tradeoff  analysis  is  less  likely  because  the  sample  size  (organizational  scope)  for  the  entire 
organizational  unit  is  four  basic  units  for  this  case  study.  An  appraisal  with  this  sample  size  is 
generally  thought  of  as  a  reasonable  sized  appraisal  given  that  it  can  be  for  the  entire  company. 
Refer  to  Case  Study  1  and  2  for  examples  of  how  this  type  of  tradeoff  analysis  could  be 
performed. 
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Applying  the  Data  Coverage  Requirements 

After  interviewing  Green  Company  management,  process  and  quality  personnel,  the  following 
support  functions  were  found  to  exist  in  the  organizational  unit 

•  Engineering  Process  Group 

•  Configuration/Release  Management 

•  Quality  Assurance 

•  Measurement  Specialists 

•  Procurement  and  Training 

The  table  below  shows  the  mapping  of  process  areas  to  the  basic  units  and  support  functions  that 
perform  the  related  work.  This  is  a  level  of  detail  not  specifically  illustrated  in  the  previous  case 
studies,  though  obviously  this  type  of  understanding  is  necessary  to  support  data  collection 
planning  in  all  cases. 

Table  40:  Process  Areas  and  Basic  Units/Support  Functions 


Process  Area  Name 

Work  Done  By... 

Requirements  Management 

Basic  Units:  Releases  and  Special  Projects 

Project  Planning 

Basic  Units:  Releases  and  Special  Projects 

Project  Monitoring  and  Control 

Basic  Units:  Releases  and  Special  Projects 

Supplier  Agreement  Management 

Support  Function:  Procurement 

Measurement  and  Analysis 

Support  Function:  Measurement  Specialists 

Process  and  Product  Quality  Assurance 

Support  Function:  Quality  Assurance 

Configuration  Management 

Support  Function:  Configuration/Release  Management 

Requirements  Development 

Basic  Units:  Releases  and  Special  Projects 

Technical  Solution 

Basic  Units:  Releases  and  Special  Projects 

Product  Integration 

Basic  Units:  Releases  and  Special  Projects 

Verification 

Basic  Units:  Releases  and  Special  Projects 

Validation 

Basic  Units:  Releases  and  Special  Projects 

Organizational  Process  Focus 

Support  Function:  Engineering  Process  Group 

Organizational  Process  Definition 

Support  Function:  Engineering  Process  Group 

Organizational  Training 

Support  Function:  Procurement 

Integrated  Project  Management 

Basic  Units:  Releases  and  Special  Projects 

Risk  Management 

Basic  Units:  Releases  and  Special  Projects 

Decision  Analysis  and  Resolution 

Basic  Units:  Releases  and  Special  Projects 
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Data  collection  planning  for  each  process  area  in  scope  of  the  appraisal  is  driven  by  the  data 
coverage  rules  found  in  activity  2.4.1,  Verify  Objective  Evidence.  A  high-level  view  of  the  data 
coverage  requirements  for  this  scenario  appears  in  the  matrices  below.  In  each  matrix,  “ART” 
stands  for  artifacts  and  “AFF”  stands  for  affirmations.  Cells  that  are  not  shaded  in  grey  represent 
areas  where  data  collection  would  be  required.  Note,  there  are  alternative  allowable 
configurations,  but  the  number  of  cells  in  the  matrices  depicts  the  data  coverage  requirements. 

Table  41:  Basic  Unit/Support  Function  versus  Process  Area  Map 


REQM 

pp 

PMC 

RD 

TS 

PI 

VER 

VAL 

IPM 

Relevant  Sampling  Factors 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Flagship  Releases 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

New  Division  A  Releases 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

New  Division  B  Releases 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Data  Warehouse  201 1 

0 

X 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

SecurityReporting  ABC 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Firmware  Tester  201 1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Procurement 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Measurement  Specialist 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Quality  Assurance 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Configuration  Management 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Engineering  Process  Group 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

3 

4 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 
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Table  42:  Basic  Unit/Support  Function  versus  Process  Area  Map  (continued) 


RSKM 

DAR 

SAM 

MA 

PPQA 

CM 

OPF 

OPD 

OT 

Relevant  Sampling  Factors 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Art 

Aff 

Flagship  Releases 

X 

X 

X 

X 

X 

X 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

New  Division  A  Releases 

X 

X 

X 

X 

X 

X 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

New  Division  B  Releases 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Data  Warehouse  201 1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

SecurityReporting  ABC 

X 

X 

X 

X 

X 

X 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Firmware  Tester  201 1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Procurement 

0 

0 

0 

0 

X 

X 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

X 

X 

Measurement  Specialist 

0 

0 

0 

0 

0 

0 

X 

X 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Quality  Assurance 

0 

0 

0 

0 

0 

0 

0 

0 

X 

X 

0 

0 

0 

0 

0 

0 

0 

0 

Configuration  Management 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

X 

X 

0 

0 

0 

0 

0 

0 

Engineering  Process  Group 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

X 

X 

X 

X 

0 

0 

3 

3 

3 

3 

3 

3 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

Notes: 

•  The  single  basic  unit  for  the  first  subgroup  (Flagship  Division  System  Releases)  supplies  both 
artifacts  and  affirmations  for  all  process  areas  applicable  to  basic  units.  This  is  in  accordance 
with  the  data  in  Coverage  Rule  1  for  basic  units. 

•  In  the  next  subgroup  (Releases  in  New  Division  A  or  B)  only  one  of  the  two  basic  units  is 
sampled.  In  accordance  with  Coverage  Rule  1  for  basic  units,  artifacts  and  affirmations  are 
supplied  for  all  process  areas  applicable  to  basic  units  here  as  well. 

•  The  next  subgroup  consists  of  three  basic  units  (DataWarehouse201 1,  SecurityReportingABC 
and  FirmwareTester201 1),  and  only  two  of  these  are  sampled.  In  this  subgroup.  Coverage 
Rule  1  for  basic  units  is  satisfied  by  the  fact  that  “SecurityReportingABC”  provides  artifacts 
and  affirmations  for  all  process  areas  applicable  to  basic  units.  Due  to  the  small  size  of  the 
sample,  this  also  meets  the  data  coverage  rule  Coverage  Rule  2  for  basic  units.  According  to 
Coverage  Rule  3  for  basic  units,  additional  data  (artifacts  OR  affirmations)  must  be  provided 
by  the  second  sampled  basic  unit  for  at  least  one  process  area  —  here  the  plan  exceeds  this 
minimum  by  seeking  affirmations  for  the  basic  unit  “DataWarehouse2011”  on  a  number  of 
process  areas  (including  RD,  TS,  PI,  VER  and  VAL). 
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Case  3  (Small  Development  Organization)  Summary 

Under  the  rules  of  SCAMPI  VI. 3,  this  company  will  be  able  to  explore  many  different 
alternatives  pertaining  to  the  definition  of  the  organizational  unit  and  associated  organizational 
scope,  as  appropriate.  At  this  point,  the  lead  appraiser  and  representatives  from  the  organization 
can  establish  reliable  estimates  for  the  detailed  data  collection  activities  that  follow,  and  consider 
a  variety  of  trade-offs  in  the  scope  and  sample  of  the  appraisal.  Based  on  ‘sanity  checks’  looking 
at  the  variation  in  process  implementation  (perhaps  through  examination  of  PPQA  records)  the 
adequacy  of  the  above  minimum  sample  can  be  evaluated.  Costs  associated  with  alternative 
scenarios  that  result  from  reconsidering  other  assumptions  can  also  be  evaluated. 


Summary  of  Case  Studies 

Many  different  permutations  of  the  organizational  unit  are  supported  by  SCAMPI  VI. 3. 

The  analysis  of  sampling  factors,  and  examination  of  sampling  trade-offs,  helps  the  appraisal  team 
leader  to  work  with  the  appraisal  sponsor  to  seek  the  most  efficient  appraisal  for  a  given  scope  of 
the  organizational  unit.  If  the  magnitude  of  the  organizational  scope  of  the  appraisal  exceeds  the 
constraints  of  the  sponsor,  then  eliminating  one  or  another  source  of  diversity  in  the 
organizational  unit  can  provide  needed  de-scoping. 
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Appendix  G  SCAMPI  A  Appraisals  Including  Multiple  Models 


Purpose 

The  purpose  of  this  appendix  is  to  provide  guidance  for  SCAMPI  As  that  incorporate  more  than 
one  reference  model  (e.g.,  a  CMMI  constellation  with  the  People  CMM).  The  first  area  for 
guidance  that  sets  the  stage  and  underlies  all  of  the  other  considerations  is  scoping.  When  scoping 
an  appraisal  involving  more  than  one  reference  model  the  lead  appraiser  needs  to  remember  that 
the  requirements  for  each  reference  model  included  in  the  appraisal  scope  must  be  satisfied.  The 
description  of  the  organizational  unit  needs  to  be  specified  uniquely  for  each  reference  model. 
There  may  be  overlap,  at  times  substantial,  among  the  organizational  units  for  the  reference 
models.  This  is  necessary  because  some  organizations  may  have  very  tightly  integrated  basic  units 
that  implement  processes  that  span  multiple  reference  models  (e.g.,  configuration  management), 
while  others  may  have  sister  organizations  that  fall  under  the  same  management  structure  but 
implement  different  process  sets  with  only  limited  overlap  (e.g.,  they  share  infrastructure,  but  use 
very  different  business  practices). 

Single  Event 

An  appraisal  addressing  more  than  one  reference  model  is  considered  to  be  a  single  appraisal  from 
an  appraisal-planning  and  record-keeping  perspective.  The  planning  will  cover  all  reference 
models  involved  with  a  single  appraisal  plan  (or  single  collection  of  planning  documents.  A  single 
SAS  entry  and  one  appraisal  disclosure  statement  are  to  be  generated.  A  single  data  package  will 
be  submitted  to  the  SEI  upon  completion  of  the  appraisal.  There  will  be  one  lead  appraiser.  The 
lead  appraiser  and  appraisal  team  members  will  be  recognized  as  having  participated  in  a  single 
appraisal  for  the  purpose  of  satisfying  pre-requisites  to  attend  training  or  renewing  credentials. 
Conducting  an  appraisal  of  more  than  one  reference  model  at  a  single  event  will  save  some  costs 
over  having  separate  appraisal  teams  and  separate  events  (e.g.,  common  appraisal  planning,  team 
training,  preparation  and  conduct). 

Planning 

Overview 

A  key  step  to  planning  the  appraisal  is  analyzing  the  organizational  structure  to  determine  the 
appropriate  organizational  unit  for  each  of  the  included  reference  model.  During  this  analysis  the 
appraisal  team  leader  in  consultation  with  the  sponsor  determines  which  business  objectives  are 
appropriate  to  each  reference  model  and  how  these  align  with  the  appraisal  objectives.  The 
appraisal  team  leader  will  then  work  with  the  sponsor  to  identify  and  document  the  organizational 
unit  and  organizational  scope  for  each  reference  model  through  selection  of  the  basic  units  and 
groups  providing  support  functions  that  will  participate  in  the  appraisal.  Throughout  the  planning 
process  various  tailoring  options  for  the  different  reference  models  will  be  exercised  and 
documented. 
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An  integrated  organizational  process  architecture  will  provide  opportunities  for  appraisal 
optimization.  Some  efficiency  may  be  achieved  in  the  data  collection  for  process  management  and 
possibly  support  process  areas  based  on  the  process  architecture  structure  for  the  organization. 

Three  Organization  Types 

This  high-level  example  will  consider  three  organizations  that  are  performing  maintenance  and 
help  desk  functions.  All  three  of  the  organizations  are  using  CMMI-  DEV  for  the  maintenance 
processes  and  CMMI-SVC  for  the  help  desk  processes. 

Organization  A:  Organization  A  has  closely  integrated  maintenance  and  help  desk  functions  with 
each  team  providing  both  maintenance  and  help  desk  support  for  one  or  more  customers.  There 
will  be  an  almost  complete  overlap  between  the  organizational  unit  for  CMMI-DEV  and  the 
organizational  unit  for  CMMI-SVC. 

Organization  B:  Organization  B  has  a  centralized  help  desk  team  that  handles  all  customers.  It  has 
maintenance  teams  either  for  each  customer  or  set  of  customers  (some  of  the  maintenance  is  for 
custom  systems  and  some  is  for  product  lines).  There  will  be  little  to  no  overlap  between  the 
organizational  units  between  CMMI-DEV  and  CMMI-SVC  functions. 

Organization  C:  Organization  C  is  a  hybrid  of  Organization  A  and  Organization  B  with  a  mix  of 
centralized  and  dispersed  functions  depending  upon  the  product  line.  Some  product  lines  keep  the 
maintenance  and  help  desk  functions  separated  and  some  have  them  intermingled,  depending 
upon  which  seems  most  appropriate  for  that  particular  product  line. 

There  are  many  other  options  with  varying  degrees  of  centralization  versus  dispersal.  One  aspect 
in  analyzing  the  organization  for  planning  the  appraisal  is  to  understand  the  interfaces  between  the 
processes  and  the  organizational  structure  and  use  that  understanding  to  determine  the  basic  units. 

This  will  then  determine  the  organizational  scoping,  data  collection,  and  practice  characterizations 
for  the  organization.  Selection  of  basic  units  and  support  functions  during  the  appraisal  planning 
process  begins  the  organizational  scoping  process.  The  sections  below  describe  the  impacts  of 
having  more  than  one  reference  model  in  an  appraisal. 

Scoping 

The  description  of  the  organizational  unit  as  well  as  the  description  of  the  organizational  scope 
must  be  specified  uniquely  for  each  reference  model.  The  identification  of  the  ‘basic  units’  to  be 
sampled  may  differ  depending  upon  the  reference  models  in  scope,  and  must  be  documented  for 
each.  For  example,  CMMI-DEV  and  CMMI-ACQ  have  “projects”  while  CMMI-SVC  has  “work 
units”  and  People  CMM  has  “units”.  The  appraisal  team  leader  needs  to  evaluate  the 
characteristics  of  the  basic  units  to  understand  how  processes  may  be  implemented  differently 
across  basic  units.  For  example  if  the  basic  unit  is  executing  both  DEV  and  SVC  based  processes, 
do  we  need  to  examine  it  from  both  perspectives  or  are  the  implementations  similar  enough  that 
they  can  be  treated  as  one? 

Additionally,  there  may  be  support  functions  that  exist  and  relate  to  multiple  reference  models,  or 
may  be  defined  in  reference  model-specific  structures  or  a  combination  of  both.  For  example,  for 
process  management  process  areas,  an  organization  may  have  one  process  group  for  all  reference 
models,  or  an  over-arching  process  group  followed  by  reference  model  specific  process  groups,  or 
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structure  process  groups  along  organization  lines  (as  some  organization  currently  do  with 
division-specific  process  groups). 

Sampling  Factors 

A  key  consideration  in  scoping  is  analyzing  sampling  factors  that  affect  the  way  people  do  the 
work  in  different  basic  units  for  each  organizational  unit  defined  for  each  reference  model.  Some 
sampling  factors  will  have  the  same  implication  for  all  reference  models  while  other  sampling 
factors  may  have  reference  model-specific  implications,  and  not  be  relevant  to  all  reference 
models  in  scope.  This  analysis  is  used  to  form  a  representative  sample  of  the  organizational  unit  to 
identify  the  organizational  scope  of  the  appraisal.  The  analysis  of  the  sampling  factors  needs  to  be 
documented  clearly  for  each  reference  model  since  they  may 

•  be  different  for  each  reference  model 

•  have  the  same  implication  for  all  reference  models 

•  have  reference  model-specific  implications,  and  not  be  relevant  to  all  reference  models  in 
scope 

Subgroups  and  Basic  Units 

Next,  subgroups  are  defined  for  each  organizational  unit  based  on  the  analysis  of  the  sampling 
factors.  Subgroups  defined  using  the  sampling  factors  represent  clusters  of  similar  process 
implementations.  Basic  units  within  these  subgroups  are  sampled  to  form  the  organizational  scope 
for  each  organizational  unit.  If  there  is  overlap  in  the  organizational  units  defined  for  each 
reference  model  there  may  be  overlap  in  the  organizational  scope  for  each  reference  model  as 
well,  depending  on  how  work  is  performed  by  different  basic  units.  Subgroups  that  are  the  basis 
for  the  final  sampling  decisions  may  take  on  different  forms  such  as: 

•  Reference  model-specific  subgroups  (e.g.,  Service  Level  Agreement  (SLA)-based  services 
within  a  given  service  category  vs.  non-SLA-based  services  in  that  same  service  category) 

•  Common  subgroups  (e.g.,  Measurement  and  Analysis  staff  who  support  all  government  work 
(e.g..  Development  (DEV)  and  Services  (SVC)  related  work) 

Process  Area  Mapping 

The  business  organization  and  organization  structure  determine  the  extent  to  which  data 
sufficiency  (process  area  mapping)  analysis  can  be  optimized.  Mapping  of  process  areas  to  basic 
units  and  support  functions  is  established  to  support  data  collection  planning  for  each 
organizational  unit  for  each  reference  model  within  the  appraisal  scope.  This  mapping  varies 
depending  on  the  way  processes  are  implemented  in  each  organizational  unit.  For  core  process 
areas,  sampling  of  objective  evidence  is  needed  for  each  reference  model  where  implementation 
of  these  processes  is  unique  to  each  reference  model.  Some  organizations  however,  may  share 
resources  to  perform  support  functions  like  Configuration  Management  (CM)  across  both 
development  and  services  activities.  In  this  instance,  mapping  of  the  core  process  area  CM  is  at 
the  organizational  level  for  both  reference  models.  In  this  case,  objective  evidence  may  be  reused 
to  support  each  practice  for  each  reference  model  for  CM  since  CM  resources  support  both 


CMU/SEI-201 1-HB-001  |  243 


development  and  services  efforts.  Other  organizations  may  establish  separate  Configuration 
Management  functions  for  development  and  services  efforts.  While  still  performed  at  the 
organizational  level  for  each  reference  model,  the  mapping  of  CM  would  be  unique  for  each 
organizational  unit.  Reference  model  unique  process  areas  may  be  instantiated  in  each  and  every 
basic  unit  and  therefore  objective  evidence  would  be  collected  uniquely  for  each  process  area.  It  is 
also  possible  that  a  given  process  area  is  instantiated  in  parallel  support  functions  that  exist  within 
given  subgroups,  or  are  shared  across  a  number  of  subgroups.  The  mapping  of  process  areas  to 
basic  units  and  support  functions  will  show  how  each  reference  model  scope  relates  to  each 
organizational  scope.  The  appraisal  team  leader  needs  to  evaluate  and  determines  the  process  area 
mapping  applicable  to  the  organizational  scope  of  the  appraisal  working  closely  with  the  sponsor. 

Data  Collection  Planning 

The  data  collection  plan  will  document  which  artifacts  and  affirmations  apply  to  each  practice,  in 
each  process  area,  in  each  reference  model.  Differences  in  the  way  people  are  organized  to 
perform  the  work  within  each  organizational  unit  for  each  reference  model  affects  the  sources  of 
data  which  must  be  considered  to  appraise  each  process  area  depending  on  the  analysis  of  the 
process  area  mapping.  Approaches  for  collection  of  shared  data  across  reference  models  (e.g., 
common  affirmations  and  artifacts)  are  addressed  in  detail  in  the  data  collection  plan.  Planning  for 
common  affirmations  (e.g.,  interviews,  demonstrations,  and  presentations)  across  organizational 
units  when  common  processes  are  used  across  each  reference  model  will  improve  the  efficiency 
of  the  Conduct  Appraisal  phase.  Care  must  be  taken  to  balance  the  use  of  common  affirmations 
and  artifacts  to  ensure  that  sufficient  objective  evidence  is  collected  to  verify  implementation  of 
each  practice  in  each  reference  model  within  the  appraisal  scope  based  on  the  process  area 
mapping.  This  requires  a  detailed  understanding  of  how  processes  are  implemented  in  each 
organizational  unit  for  each  reference  model. 

Risk  Management 

Examples  of  risks  specific  to  appraisals  of  more  than  one  reference  models  include: 

•  Team  size  and  work  load  imbalance 

•  Managing  a  larger  team  with  extended  period  of  on-site  phase 

•  Managing  the  individual  experience  level  of  appraisal  team  members  specific  to  each 
reference  model  affecting  the  team’s  ability  to  reach  consensus 

•  Managing  the  data  collection  planning  (e.g.,  determining  objective  evidence  needed  to 
support  core  process  areas  vs.  reference  model  unique  process  areas  or  common 
implementation  of  support  functions  across  more  than  one  reference  model) 

•  Inefficient  or  incorrect  reuse  of  objective  evidence  to  support  practices  across  more  than  one 
reference  model 

•  Unclear  appraisal  objectives  due  to  multiple  sponsors  with  different  business  objectives  and 
maturity  levels 
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Appraisal  Team 

Team  Training 

In  order  to  support  team  consensus,  each  and  every  appraisal  team  member  must  meet  the 
requirements  associated  with  each  and  every  reference  model  included  in  the  scope  of  the 
appraisal.  The  appraisal  team  leader  must  be  certified  in  each  reference  model  (except  for  People- 
CMM)  in  the  scope  of  the  appraisal.  At  the  discretion  of  the  lead  appraiser  individuals  who  fall 
short  of  the  minimum  requirements  can  participate  as  long  as  the  team  meets  the  overall 
requirements.  Enough  reference  model-specific  experience  for  each  mini-team  is  needed  to 
properly  judge  reference  model  compliance  and  adequacy  of  objective  evidence  to  demonstrate 
practice  instantiations. 

Teams,  Sub-teams,  and  Mini-Teams 

Team  size  may  increase  as  compared  to  an  appraisal  of  one  reference  model  to  meet  the  increased 
training  and  expertise  requirements  as  well  as  the  increase  of  model  scope.  During  the  planning  of 
the  appraisal,  the  lead  appraiser  (in  collaboration  with  the  sponsor)  will  explicitly  document  the 
appraisal  team  structure.  Some  of  the  options  are: 

•  Identify  reference  model-specific  sub-teams  —  with  further  assignment  of  mini-teams  to 
organizational  entities  or  groups  of  process  areas 

•  Identify  mini-teams  assigned  to  organizational  entities  or  groups  of  process  areas,  without 
differentiating  reference  models 

•  Use  neither  sub-teams  nor  mini-teams,  in  which  case  the  entire  team  works  as  a  whole  on  all 
organizational  entities  and  process  areas  (though  this  is  not  very  common) 

Each  of  these  options  have  advantages  and  disadvantages  and  the  appraisal  team  leader  needs  to 
take  into  consideration  the  experience  levels  of  the  team  members  in  the  domain  areas,  assigned 
process  areas,  appraisal  experience,  etc.  The  more  people  on  each  team,  the  longer  it  takes  to  build 
consensus,  however  it  means  more  individuals  are  examining  and  judging  the  data  which  tends  to 
lead  to  more  thorough  results 

Examining  Objective  Evidence 

Depending  on  the  differences  or  similarities  in  the  way  processes  are  implemented  for  each 
reference  model,  one  piece  of  objective  evidence  may  support  more  than  one  practice  across  more 
than  one  reference  model.  The  details  of  this  requirement  are  provided  in  the  Data  Sufficiency 
section.  The  lead  appraiser  will  analyze  and  document  the  result  in  the  plan. 

Verification 

Data  sufficiency 

Data  sufficiency  rules  shall  be  applied  to  each  practice  within  each  included  reference  model 
which  will  in  some  cases  require: 

•  Reference  model-unique  data  (e.g.,  SCON  in  CMMI-SVC  must  have  enough  data  from  all 
work  groups  included) 
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•  Common  organizational  data  that  may  be  evaluated  as  it  applies  to  each  reference  model  in  turn 
(e.g.,  OPF  evidence  for  a  common  implementation  that  spans  multiple  reference  models) 

•  ‘parallel  data  sets’  for  reference  model-specific  implementations  of  process  areas  (e.g.,  data  used 
to  evaluate  Risk  Management  for  CMMI-DEV  could  be  a  completely  different  data  set  than  what 
the  team  examines  regarding  Risk  Management  for  CMMI-SVC  -  there  must  be  sufficient  data  for 
each  implementation  of  Risk  Management) 

Practice  Characterizations 

Each  practice  is  characterized  separately  for  each  reference  model,  with  the  possible  exception  of 
common  process  areas.  For  core  process  areas  that  are  implemented  at  the  organizational  level  for 
each  reference  model,  the  characterizations  could  be  same  when  based  on  the  same  objective 
evidence.  For  example,  when  shared  resources  support  Configuration  Management  across  the  defined 
organizational  units  and  their  process  implementation  is  the  same  the  resulting  characterizations  will 
likely  be  the  same  for  each  reference  model.  When  core  process  areas  are  implemented  differently  for 
each  reference  model,  objective  evidence  is  assessed  for  each  unique  implementation  of  the  practice 
and  characterizations  are  unique  to  the  organizational  unit.  This  could  result  in  a  characterization  of 
Partially  Implemented  (PI)  for  Project  Planning  (PP)  for  CMMI-  DEV  while  Fully  Implemented  (FI) 
for  Project  Planning  (PP)  for  CMMI-SVC.  Reference  model  unique  process  areas  will  result  in  unique 
characterizations.  Process  management  process  areas  common  implementation  that  spans  multiple 
reference  models  would  likely  result  in  single  characterization  of  model  practices  for  each  reference 
model. 

Data  Validation 

Multiple  validation  activities  can  be  conducted  for  reference  model  specific  findings  depending  on 
how  the  work  is  organized  across  each  organizational  unit  within  the  appraisal  scope. 

Ratings 

Separate  goal  ratings  (and  process  area  or  maturity  level  ratings,  if  produced)  are  required  for  each 
reference  model  in  the  appraisal.  There  is  no  mechanism  provided  for  deriving  a  combined  rating 
(combined  ratings  are  explicitly  prohibited).  This  does  not  prohibit  a  continuous  appraisal  from  being 
performed  and  then  equivalent  staging  being  applied  to  the  selected  organizational  unit  for  each 
CMMI  reference  model  separately.  For  core  process  areas  that  are  implemented  at  the  organizational 
level  for  each  reference  model,  the  ratings  could  be  common  when  based  on  the  same  objective 
evidence. 

Findings 

There  may  be  an  integrated  final  finding  or  multiple  final  findings  (a  tailoring  option).  All  findings 
shall  be  reviewed  and  affirmed  using  a  consensus  decision  process  involving  the  entire  appraisal  team. 
Findings  may  be  generated  separately  for  core  process  areas  for  each  CMMI  reference  model  in  scope. 
Separate  findings  may  be  beneficial  when  there  is  limited  overlap  in  organizational  units  for  each 
reference  model.  Combined  findings  would  be  appropriate  for  core  process  areas  especially  when 
there  is  significant  overlap  in  organizational  units  for  each  reference  model.  If  integrated  findings  are 
provided,  the  findings  must  be  traceable  to  each  reference  model  within  the  appraisal  scope. 
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Appendix  H  SCAMPI  A  Tailoring  Checklist 


Table  43:  Tailoring  Checklist 


MDD  Process 

Tailoring  Options 

Option 

Chosen 

(Y/N) 

Tailoring  Rationale 
(Provide  a  brief  explanation  if 
this  option  was  chosen) 

1.1.1  Determine 
Appraisal 

Objectives 

Appraisal  usage  mode 

Internal  Process  Improvement, 
Supplier  Selection,  or  Process 
Monitoring 

1.1.2  Determine 

Data  Collection 
Strategy 

Collection  of  objective  evidence 
during  the  planning  or  conduct 
phase  or  combinations  of  both 
approaches 

Discovery,  managed  discovery, 
verification 

1.1.4  Determine 
Appraisal  Scope 

Use  of  incremental  appraisals 

1.1.5  Determine 
Appraisal  Outputs 

Selection  of  optional  ratings  and 
findings 

Optional  findings:  non-model 
findings,  final  report, 
recommendations,  process 
improvement  action  plan,  basic 
unit  specific  findings 

Optional  ratings:  process  area, 
maturity  level, 

1.3.2  Prepare 

Team 

Deliver  Appraisal  Method  training 
to  more  than  a  single  team  at  the 
same  event 

Reference  waiver 

1.3.2  Select  Team 

Members 

Accept  one  team  member  with  no 
field  experience 

1.3.4  Prepare 

Team 

Teams  involved  in  U.S. 
government  source  selection  or 
process  monitoring 

Specify  additional  training  to  be 
provided 

1.4.2  Inventory 
Objective  Evidence 

Collection  of  objective  evidence 
during  the  planning  or  conduct 
phase  or  combinations  of  both 
approaches 

Discovery,  managed  discovery, 
verification 

1.5.1  Perform 

Readiness  Review 

Practice  characterizations  are  done 
before  or  during  the  readiness 
review 

Conduct  Appraisal  phase  begins 

1.5.1  Perform 

Readiness  Review 

Class  B  or  C  (SCAMPI  B  or  C) 
used  as  readiness  review 

Conduct  Appraisal  phase  begins 
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MDD  Process 

Tailoring  Options 

Option 

Chosen 

(Y/N) 

Tailoring  Rationale 
(Provide  a  brief  explanation  if 
this  option  was  chosen) 

1.5.1  Perform 

Readiness  Review 

Used  as  one  of  the  iterative  data 
collection  activities  in  a  Managed 
Discovery  approach. 

2.2.1  Examine 
Objective  Evidence 
from  Artifacts 

Evaluate  the  content  of  artifacts  to 
determine  how  it  supports  model 
practice  implementation  during 
readiness  review  or  other  appraisal 
preparation  activity 

Conduct  Appraisal  phase  begins 

2.2.2  Examine 
Objective  Evidence 
from  Affirmations 

Evaluate  the  information  from 

affirmations  to  determine  how  it 
supports  model  practice 
implementation  during  readiness 
review  or  other  appraisal 
preparation  activity 

Conduct  Appraisal  phase  begins 

2.2.2  Examine 
Objective  Evidence 
from  Affirmations 

Use  of  virtual  methods  for 

affirmations 

Risk  mitigation  plan 

2.6.2  Determine 

Process  Area 

Ratings 

This  is  an  optional  activity  selected 
at  the  discretion  of  the  appraisal 
sponsor 

2.6.3  Determine 

Process  Area 

Profile 

This  is  an  optional  activity  selected 
at  the  discretion  of  the  appraisal 
sponsor 

2.6.4  Determine 
Maturity  Level 

This  is  an  optional  activity  selected 
at  the  discretion  of  the  appraisal 
sponsor 

3.1.2  Conduct 

Executive 

Session(s) 

This  is  an  optional  activity  selected 
at  the  discretion  of  the  appraisal 
sponsor 

3.1.3  Plan  for  Next 
Steps 

This  is  an  optional  activity  selected 
at  the  discretion  of  the  appraisal 
sponsor 

3.2.1  Collect 

Lessons  Learned 

This  is  an  optional  activity  selected 
at  the  discretion  of  the  appraisal 
sponsor 

CMU/SEI-201 1-HB-001  |  248 


References/Bibliography 


[SEI  2002] 

Barbour,  Rick;  Benhoff,  Melanie;  Gallagher,  Brian;  Eslinger,  Suellen;  Bernard,  Thomas;  Ming, 
Lisa;  Rosa,  Linda;  Ryan,  Charlie.  Standard  CMMI®  Appraisal  Method  for  Process  Improvement 
(SCAMP ISM),  Version  1.1:  Method  Implementation  Guidance  for  Government  Source  Selection 
and  Contract  Process  Monitoring  (CMU/SEI-2002-I  IB-002).  Pittsburgh,  PA:  Software 
Engineering  Institute,  Carnegie  Mellon  University,  September  2002. 
http://www.sei.cmu.edu/library/abstracts/reports/02hb002.cfm 

[SEI  2004] 

Dr.  Behrens,  Sandi;  Mogilensky,  Judah;  Masters,  Steve.  CMMI®-Based Professional 
Certifications:  The  Competency  Lifecycle  Framework.  Pittsburgh,  PA:  Software  Engineering 
Institute,  Carnegie  Mellon  University,  December,  2004. 
http  ://www.  sei  .emu.  edu/reports/04sr0 13.pdf 

[Curtis  2009] 

Curtis,  Bill;  Hefley,  William  E.;  &  Miller,  Sally  A.  The  People  CMM:  A  Framework  for  Hitman 
Capital  Management,  2nd  Edition.  Boston,  MA:  Addison-Wesley,  2009. 

[SEI  2010a] 

CMMI  Product  Team.  CMMI  for  Acquisition,  Version  1.3  (CMU/SEI-2010-TR-032).  Pittsburgh, 
PA:  Software  Engineering  Institute,  Carnegie  Mellon  University,  November  2010. 
http://www.sei.cmu.edu/library/abstracts/reports/10tr032.cfm. 

[SEI  2010b] 

CMMI  Product  Team.  CMMI  for  Development,  Version  1.3  (CMU/SEI-2010-TR-033). 

Pittsburgh,  PA:  Software  Engineering  Institute,  Carnegie  Mellon  University,  November  2010. 
http://www.sei.cmu.edu/library/abstracts/reports/10tr033.cfm. 

[SEI  2010c] 

CMMI  Product  Team.  CMMI  for  Services,  Version  1.3  (CMU/SEI-2010-TR-034).  Pittsburgh,  PA: 
Software  Engineering  Institute,  Carnegie  Mellon  University,  November  2010. 
http://www.sei.cmu.edu/library/abstracts/reports/10tr034.cfm. 

[SEI  2010d] 

Caralli,  Richard;  Allen,  Julia;  White,  David;  CERT' Resilience  Management  Model  (RMM),  A 
Maturity  Model  for  Managing  Operational  Resilience.  Upper  Saddle  River,  NJ:  Addison-Wesley; 
The  SEI  Series  in  Software  Engineering;  Software  Engineering  Institute,  Carnegie  Mellon 
University,  2011. 

[SEI  2011a] 

SCAMPI  Upgrade  Team.  Appraisal  Requirements  for  CMMI,  Version  1.2  (ARC,  VI. 3) 
(CMU/SEI-201  l-TR-001).  Pittsburgh,  PA:  Software  Engineering  Institute,  Carnegie  Mellon 
University,  expected  April  2011.  http://www.sei.cmu.edu/library/abstracts/reports/lltr001.cfm. 


CMU/SEI-201 1-HB-001  |  249 


[SEI  2011b] 

Osiecki,  Lawrence  T.;  Phillips,  Mike;  Scibilia,  John.  Understanding  and  Leveraging  a  Supplier’s 
CMMI®  Efforts:  A  Guidebook  for  Acquirers  (Revised  for  VI. 3).  Pittsburgh,  PA:  Software 
Engineering  Institute,  Carnegie  Mellon  University,  expected  March  2011. 


CMU/SEI-201 1-HB-001  |  250 


Glossary 


The  MDD  glossary  defines  many,  but  not  all,  terms  used  in  this  document.  Consult  the  following 
additional  sources  for  terms  and  definitions  supplementary  to  the  MDD  glossary: 

•  CMMI  model  glossary  and  terminology 

•  Appraisal  Requirements  for  CMMI  (ARC)  glossary 

Terms  that  are  important  in  order  to  understand  this  document  are  duplicated  from  the  model 
document  or  from  the  ARC  for  convenience. 

affirmation 

An  oral  or  written  statement  confirming  or  supporting  implementation  (or  lack  of  implementation) 
of  a  model  practice  provided  by  the  implemented  of  the  practice,  provided  via  an  interactive 
forum  in  which  the  appraisal  team  has  control  over  the  interaction. 

Examples  of  oral  affirmations  include  interview  responses,  presentations,  and  demonstrations  of  a 
tool  or  mechanism  related  to  implementation  of  a  CMMI  model  practice,  as  long  as  these 
presentations  and  demonstrations  are  provided  in  an  interactive  setting. 

Examples  of  written  affirmations  include  written  statements  provided  by  the  implemented  of  the 
practice  to  the  appraisal  team  via  an  interactive  forum  (e.g.,  email)  in  which  the  appraisal  team 
has  the  ability  to  ask  questions  either  orally  or  written.  Presentation  and  demonstration  materials 
provided  in  an  interactive  setting  to  the  appraisal  team  can  also  be  written  affirmations  if  they  are 
not  outputs  of  the  process,  in  which  case  they  could  be  artifacts  instead. 

alternative  practice 

A  practice  that  is  a  substitute  for  one  or  more  practices  contained  in  a  reference  model  that 
achieves  an  equivalent  effect  toward  satisfying  the  goal  associated  with  the  model  practice. 
Alternative  practices  are  not  necessarily  one-for-one  replacements  for  the  model  practices. 

appraisal 

An  examination  of  one  or  more  processes  by  a  trained  team  of  professionals  using  an  appraisal 
reference  model  as  the  basis  for  detennining,  as  a  minimum,  strengths  and  weaknesses. 

Appraisal  Disclosure  Statement  (ADS) 

A  summary  statement  describing  the  ratings  generated  as  outputs  of  the  appraisal,  and  the 
conditions  and  constraints  under  which  the  appraisal  was  perfonned.  The  ADS  should  be  used  for 
public  disclosures  of  maturity  level  or  capability  level  ratings  so  they  can  be  interpreted 
accurately. 

appraisal  findings 

The  results  of  an  appraisal  that  identify,  as  a  minimum,  any  strengths  and  weaknesses  within  the 
appraisal  scope.  Appraisal  findings  are  inferences  drawn  from  corroborated  objective  evidence. 
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appraisal  method  class 

A  family  of  appraisal  methods  that  satisfy  a  defined  subset  of  requirements  in  the  Appraisal 
Requirements  for  CMMI  (ARC).  These  classes  are  defined  so  as  to  align  with  typical  usage 
modes  of  appraisal  methods. 

appraisal  modes  of  usage 

The  contexts  in  which  an  appraisal  method  might  be  utilized.  Appraisal  modes  of  usage  identified 
for  the  SCAMPI  method  include  internal  process  improvement,  supplier  selection,  and  process 
monitoring. 

appraisal  objectives 

The  desired  outcome(s)  of  an  appraisal  process. 

appraisal  output 

All  of  the  tangible  results  from  an  appraisal  (see  appraisal  record). 

appraisal  participants 

Members  of  the  organizational  unit  who  participate  in  providing  information  during  the  appraisal. 

appraisal  rating 

The  value  assigned  by  an  appraisal  team  to  (a)  a  goal  or  process  area,  (b)  the  capability  level  of  a 
process  area,  or  (c)  the  maturity  level  of  an  organizational  unit.  The  rating  is  determined  by 
enacting  the  defined  rating  process  for  the  appraisal  method  being  employed. 

appraisal  record 

An  orderly,  documented  collection  of  information  that  is  pertinent  to  the  appraisal  and  adds  to  the 
understanding  and  verification  of  the  appraisal  findings  and  ratings  generated.  Provided  to  the 
sponsor  prior  to  the  report  results  phase  of  the  appraisal. 

appraisal  reference  model 

The  CMMI  or  other  model  to  which  an  appraisal  team  correlates  implemented  process  activities. 

appraisal  scope 

The  definition  of  the  boundaries  of  the  appraisal  encompassing  the  organizational  limits  and  the 
model  limits  within  which  the  processes  to  be  investigated  operate.  The  appraisal  scope  includes 
the  reference  model  scope,  organizational  unit  and  organizational  scope. 

appraisal  sponsor 

The  individual,  internal  or  external  to  the  organization  being  appraised,  who  requires  the  appraisal 
to  be  performed,  and  provides  financial  or  other  resources  to  carry  it  out. 
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appraisal  tailoring 

Selection  of  options  within  the  appraisal  method  for  use  in  a  specific  instance.  The  intent  of 
tailoring  is  to  assist  an  organization  in  aligning  application  of  the  method  with  its  business  needs 
and  objectives. 

appraisal  team  leader 

The  person  who  leads  the  activities  of  an  appraisal  and  has  satisfied  the  qualification  criteria  for 
experience,  knowledge,  and  skills  defined  by  the  appraisal  method. 

artifact 

A  tangible  form  of  objective  evidence  indicative  of  work  being  performed  that  represents  either 
the  primary  output  of  a  model  practice  or  a  consequence  of  implementing  a  model  practice. 

assessment 

An  appraisal  that  an  organization  does  internally  for  the  purposes  of  process  improvement.  The 
word  assessment  is  also  used  in  the  CMMI  Product  Suite  in  an  everyday  English  sense  (e.g.,  risk 
assessment). 

basic  unit 

A  managed  set  of  interrelated  resources  which  delivers  one  or  more  products  or  services  to  a 
customer  or  end  user  and  typically  operates  according  to  a  plan.  Such  a  plan  is  frequently 
documented  and  specifies  the  products  or  services  to  be  delivered  or  implemented,  the  resources 
and  funds  to  be  used,  the  work  to  be  done,  and  the  schedule  or  doing  the  work. 

capability  evaluation 

An  appraisal  by  a  trained  team  of  professionals  used  as  a  discriminator  to  select  suppliers,  to 
monitor  suppliers  against  the  contract,  or  to  determine  and  enforce  incentives.  Evaluations  are 
used  to  gain  insight  into  the  process  capability  of  a  supplier  organization  and  are  intended  to  help 
decision  makers  make  better  acquisition  decisions,  improve  subcontractor  performance,  and 
provide  insight  to  a  purchasing  organization. 


consensus 

A  method  of  decision  making  that  allows  team  members  to  develop  a  common  basis  of 
understanding  and  develop  general  agreement  concerning  a  decision  that  all  team  members  are 
willing  to  support. 

consolidation 

The  activity  of  collecting  and  summarizing  the  information  provided  into  a  manageable  set  of  data 
to  (a)  determine  the  extent  to  which  the  data  are  corroborated  and  cover  the  areas  being 
investigated,  (b)  determine  the  data’s  sufficiency  for  making  judgments,  and  (c)  revise  the  data- 
gathering  plan  as  necessary  to  achieve  this  sufficiency. 
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corroboration 

The  activity  of  considering  multiple  pieces  of  objective  evidence  in  support  of  a  judgment 
regarding  an  individual  CMMI  model  practice. 

coverage  criteria 

The  specific  criterion  that  must  be  satisfied  in  order  for  coverage  to  be  claimed. 

data  collection  session 

An  activity  during  which  objective  evidence  is  gathered.  Data  collection  sessions  (or  activities) 
include  artifact  reviews  and  affirmations. 

discovery-based  appraisal 

An  appraisal  in  which  limited  objective  evidence  is  provided  by  the  appraised  organization  prior 
to  the  appraisal,  and  the  appraisal  team  must  probe  and  uncover  a  majority  of  the  objective 
evidence  necessary  to  obtain  sufficient  coverage  of  reference  model  practices.  Discovery-based 
appraisals  typically  involve  substantially  greater  appraisal  team  effort  than  verification-based 
appraisals,  in  which  much  of  the  objective  evidence  is  provided  by  the  appraised  organization. 
(See  verification-based  appraisal  for  contrast.) 

document 

A  collection  of  data,  regardless  of  the  medium  on  which  it  is  recorded,  that  generally  has 
permanence  and  can  be  read  by  humans  or  machines.  Documents  can  be  work  products  reflecting 
the  implementation  of  one  or  more  model  practices.  These  documents  typically  include  work 
products  such  as  organizational  policies,  procedures,  and  implementation-level  work  products. 
Documents  may  be  available  in  hardcopy,  softcopy,  or  accessible  via  hyperlinks  in  a  web-based 
environment. 

focused  investigation 

A  technique  to  prioritize  appraisal  team  effort  based  on  the  continuous  collection  and 
consolidation  of  appraisal  data,  and  monitoring  of  progress  toward  achieving  sufficient  coverage 
of  reference  model  practices.  Appraisal  resources  are  targeted  toward  those  areas  for  which 
further  investigation  is  needed  to  collect  additional  data  or  verify  the  collected  set  of  objective 
evidence. 

fully  implemented  (FI) 

A  practice  characterization  value  assigned  to  a  process  instantiation  when  (1)  sufficient  artifacts 
are  present  and  judged  to  be  adequate,  (2)  affirmation  exists  to  confirm  the  implementation,  and 
(3)  no  weaknesses  are  noted. 

instantiation 

The  implementation  of  a  model  practice  used  in  the  appropriate  context  within  the  boundaries  of 
an  organizational  unit. 
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institutionalization 

The  ingrained  way  of  doing  business  that  an  organization  follows  routinely  as  part  of  its  corporate 
culture. 

internal  process  improvement  (IPI) 

An  appraisal  mode  of  usage  in  which  organizations  appraise  internal  processes,  generally  to  either 
baseline  their  process  capability,  to  establish  or  update  a  process  improvement  program,  or  to 
measure  progress  in  implementing  such  a  program. 

interviews 

A  meeting  of  appraisal  team  members  with  appraisal  participants  for  the  purpose  of  gathering 
information  relative  to  work  processes  in  place.  In  SCAMPI,  this  includes  face-to-face  interaction 
with  those  implementing  or  using  the  processes  within  the  organizational  unit.  Interviews  are 
typically  held  with  various  groups  or  individuals,  such  as  project  or  work  group  leaders, 
managers,  and  practitioners.  A  combination  of  formal  and  informal  interviews  may  be  held  and 
interview  scripts  or  exploratory  questions  developed  to  elicit  the  information  needed. 

largely  implemented  (LI) 

A  practice  characterization  value  assigned  to  a  process  instantiation  when  (1)  sufficient  artifacts 
are  present  and  judged  to  be  adequate,  (2)  affirmation  exists  to  confirm  the  implementation,  and 
(3)  one  or  more  weaknesses  are  noted. 

lead  appraiser 

A  person  who  has  achieved  recognition  from  an  authorizing  body  to  perform  as  an  appraisal  team 
leader  for  a  particular  appraisal  method. 

managed  discovery 

A  phased  data  collection  approach  beginning  with  an  initial  data  call  for  a  pre-detennined  set  of 
artifacts,  followed  by  a  set  of  iterative  calls  based  on  the  appraisal  team’s  evaluation  of  the  work 
products  and  remaining  evidence  gaps.  Managed  Discovery  attempts  to  balance  verification  and 
discovery  activities  by  using  a  phased  approach  to  data  collection  to  mitigate  the  risk  of  the 
organization  failing  to  provide  the  needed  data  or  providing  inappropriate  data. 

mini-team 

A  subset  of  the  appraisal  team  members,  typically  two  or  three,  assigned  primary  responsibility 
for  collection  of  sufficient  appraisal  data  to  ensure  coverage  of  their  assigned  reference  model 
process  areas  or  basic  units. 

non-model  findings 

Findings  that  have  significant  positive  or  negative  impact  on  the  enactment  of  processes  within 
the  organizational  unit  that  do  not  directly  relate  to  model  practices. 
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not  implemented  (Nl) 

A  practice  characterization  value  assigned  when  the  appraisal  team  determines  insufficient 
objective  evidence  exists  to  state  that  the  practice  is  implemented.  That  is,  artifacts  are  absent  or 
judged  to  be  inadequate,  no  other  evidence  (affirmations)  supports  the  practice  implementation, 
and  one  or  more  weaknesses  are  noted. 


not  yet  (NY) 

A  practice  characterization  value  assigned  when  the  basic  unit  has  not  yet  reached  the  phase  in  the 
lifecycle  within  the  appraisal  scope  to  have  implemented  the  practice. 

objective  evidence 

Artifacts  or  affirmations  used  as  indicators  of  the  implementation  or  institutionalization  of  model 
practices. 

organizational  scope 

The  collection  of  basic  units  and  support  functions  that  provides  instantiations  of  practices  used 
within,  and  representative  of,  an  organizational  unit. 

organizational  unit 

That  part  of  an  organization  that  is  the  subject  of  an  appraisal  and  to  which  the  appraisal  result 
will  be  generalized.  An  organizational  unit  deploys  one  or  more  processes  that  have  a  coherent 
process  context  and  operates  within  a  coherent  set  of  business  objectives.  An  organizational  unit 
is  typically  part  of  a  larger  organization,  although  in  a  small  organization,  the  organizational  unit 
may  be  the  whole  organization. 

partially  implemented  (PI) 

A  practice  characterization  value  assigned  to  a  process  instantiation  when  (1)  artifacts  are  absent 
or  judged  to  be  inadequate,  (2)  affirmations  suggest  that  some  aspects  of  the  practice  are 
implemented,  and  (3)  one  or  more  weaknesses  are  noted;  or  (1)  sufficient  artifacts  is  present  and 
judged  to  be  adequate,  (2)  no  other  evidence  (affirmations)  supports  the  artifact(s),  and  (3)  one  or 
more  weaknesses  are  noted. 

practice  characterization 

The  assignment  of  a  value  describing  the  extent  to  which  a  CMMI  model  practice  is  implemented. 
It  is  used  as  a  mechanism  to  reach  appraisal  team  consensus.  The  range  of  values  for  practice 
characterization  values  include  Fully  Implemented  (FI),  Largely  Implemented  (LI),  Partially 
Implemented  (PI),  Not  Implemented  (NI),  and  Not  Yet  (NY).  Practice  characterization  values  are 
assigned  to  each  reference  model  practice  for  each  process  instantiation  within  the  organizational 
scope,  and  aggregated  to  the  organizational  unit  level. 

preliminary  findings 

Strengths  and  weakness  statements  created  after  synthesizing  corroborated  objective  evidence. 
Preliminary  findings  are  provided  to  appraisal  participants  for  validation.  (See  also  appraisal 
findings.) 
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presentations 

In  SCAMPI,  a  source  of  objective  evidence  that  includes  information  prepared  by  the 
organization  and  delivered  visually  or  verbally  to  the  appraisal  team  to  aid  in  understanding  the 
organizational  processes  and  implementation  of  reference  model  practices.  This  typically  includes 
such  mechanisms  as  orientation  or  overview  briefings,  and  demonstrations  of  tools  or  capabilities. 

process  context 

The  set  of  factors  documented  in  the  appraisal  input  that  influences  the  judgment  and 
comparability  of  appraisal  ratings.  These  include,  but  are  not  limited  to,  (a)  the  size  of  the 
organizational  unit  to  be  appraised,  (b)  the  demographics  of  the  organizational  unit,  (c)  the 
application  domain  of  the  products  or  services,  (d)  the  size,  criticality,  and  complexity  of  the 
products  or  services,  and  (e)  the  quality  characteristics  of  the  products  or  services. 

process  monitoring 

An  appraisal  mode  of  usage  in  which  appraisals  are  used  to  monitor  process  implementation  (for 
example,  after  contract  award  by  serving  as  an  input  for  an  incentive/award  fee  decision  or  a  risk 
management  plan).  The  appraisal  results  are  used  to  help  the  sponsoring  organization  tailor  its 
contract  or  process  monitoring  efforts  by  allowing  it  to  prioritize  efforts  based  on  the  observed 
strengths  and  weaknesses  of  the  organization’s  processes.  This  usage  mode  focuses  on  a  long¬ 
term  teaming  relationship  between  the  sponsoring  organization  and  the  development  organization 
(buyer  and  supplier). 

process  profile 

The  set  of  goal  ratings  assigned  to  the  process  areas  in  the  scope  of  the  appraisal.  In  CMMI, 
“process  profile”  is  also  known  as  the  “process  area  profile.” 

rating 

See  appraisal  rating. 

recommendation 

Suggestions  to  resolve  a  weakness  or  to  propagate  a  strength 

sampling  factor 

Organizational  or  work  context  that  reflects  meaningful  differences  in  the  way  work  is  performed 
across  different  basic  units  within  the  organizational  unit.  Examples  of  sampling  factors  include 
location,  customer,  and  type  of  work. 

satisfied 

Rating  given  to  a  goal  when  the  aggregate  of  validated  findings  does  not  negatively  impact 
achievement  of  the  goal.  Rating  given  to  a  process  area  in  the  staged  representation  of  the 
reference  model  when  all  of  its  goals  are  rated  “satisfied.” 

strength 

Exemplary  or  noteworthy  implementation  of  a  reference  model  practice 


CMU/SEI-201 1-HB-001  |  257 


subgroup 

Cluster  of  basic  units  that  share  common  sampling  factor  alternatives  and  exhibit  similar  process 
implementations. 

sufficient  data  coverage 

A  determination  that  the  coverage  requirements  have  been  met.  See  coverage  and  coverage 
criteria. 


supplier  selection 

An  appraisal  mode  of  usage  in  which  appraisal  results  are  used  as  a  high  value  discriminator  to 
select  suppliers.  The  results  are  used  in  characterizing  the  process-related  risk  of  awarding  a 
contract  to  a  supplier. 

support  function 

An  organizational  group  that  provides  products  and/or  services  for  a  bounded  set  of  activities 
needed  by  other  portions  of  the  organization.  Examples  of  support  functions  include  a 
configuration  management  group  or  an  engineering  process  group. 

tailoring 

See  appraisal  tailoring. 

verification-based  appraisal 

An  appraisal  in  which  the  focus  of  the  appraisal  team  is  on  verifying  the  set  of  objective  evidence 
provided  by  the  appraised  organization  in  advance  of  the  appraisal,  in  order  to  reduce  the  amount 
of  probing  and  discovery  of  objective  evidence  during  the  appraisal  on-site  period.  (See 
discoveiy-based  appraisal  for  contrast.) 

weakness 

The  ineffective,  or  lack  of,  implementation  of  one  or  more  reference  model  practices. 
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